By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-05-30 10:44:42,031 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(e2a6893a-bcd2-469c-a16f-e06fbf7c4bc2, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-05-30 10:44:46,008 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(03a148fd-0bb3-4bd5-90cb-43db85f59527, queue=awx_private_queue)
2021-05-30 10:44:46,044 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(1f091c7b-e431-41b0-9317-2772d32af97c, queue=awx_private_queue)
Creating instance group tower
(changed: True)
/usr/bin/launch_awx_task.sh: line 27: unset: `#!/bin/sh': not a valid identifier
2021-05-30 10:44:46,878 INFO RPC interface 'supervisor' initialized
2021-05-30 10:44:46,878 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-05-30 10:44:46,879 INFO supervisord started with pid 150
2021-05-30 10:44:47,883 INFO spawned: 'awx-config-watcher' with pid 153
2021-05-30 10:44:47,887 INFO spawned: 'dispatcher' with pid 154
2021-05-30 10:44:47,890 INFO spawned: 'callback-receiver' with pid 155
2021-05-30 10:44:47,896 INFO spawned: 'channels-worker' with pid 156
READY
2021-05-30 10:44:49,016 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-30 10:44:49,016 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-30 10:44:49,016 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-30 10:44:49,016 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-30 10:44:51,246 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-05-30 10:44:51,246 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-05-30 10:44:51,253 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-05-30 10:44:51,253 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-05-30 10:44:51,261 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-05-30 10:44:51,261 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:177
2021-05-30 10:44:51,269 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-05-30 10:44:51,269 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-05-30 10:44:51,322 - INFO - runworker - Using single-threaded worker.
2021-05-30 10:44:51,322 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-05-30 10:44:51,323 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-05-30 10:44:51,432 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-05-30 10:44:51,436 WARNING awx.main.dispatch scaling up worker pid:180
2021-05-30 10:44:51,450 WARNING awx.main.dispatch scaling up worker pid:181
2021-05-30 10:44:51,475 WARNING awx.main.dispatch scaling up worker pid:182
2021-05-30 10:44:51,489 WARNING awx.main.dispatch scaling up worker pid:183
2021-05-30 10:44:51,496 DEBUG awx.main.tasks Syncing Schedules
2021-05-30 10:44:51,691 DEBUG awx.main.tasks Waited 0.0007815361022949219 seconds to obtain lock name: cluster_policy_lock
2021-05-30 10:44:51,705 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-05-30 10:44:51,705 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-05-30 10:44:51,705 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-05-30 10:44:51,711 DEBUG awx.main.tasks Cluster policy computation finished in 0.019073963165283203 seconds
2021-05-30 10:44:51,712 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:44:51,751 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
RESULT 2
OKREADY
2021-05-30 10:45:11,611 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1f8ae5da-055b-4580-8518-bb8661417e63, queue=awx_private_queue)
[2021-05-30 10:45:11,611: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1f8ae5da-055b-4580-8518-bb8661417e63, queue=awx_private_queue)
2021-05-30 10:45:11,633 DEBUG awx.main.dispatch delivered 1f8ae5da-055b-4580-8518-bb8661417e63 to worker[183] qsize 0
2021-05-30 10:45:11,637 DEBUG awx.main.dispatch task 1f8ae5da-055b-4580-8518-bb8661417e63 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:45:11,640 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:45:11,662 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:45:11,715 DEBUG awx.main.dispatch task 1f8ae5da-055b-4580-8518-bb8661417e63 is finished
2021-05-30 10:45:21,583 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e99e69e8-e3bc-4d6d-b006-8cdb50510d59, queue=awx_private_queue)
[2021-05-30 10:45:21,583: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e99e69e8-e3bc-4d6d-b006-8cdb50510d59, queue=awx_private_queue)
2021-05-30 10:45:21,607 DEBUG awx.main.dispatch delivered e99e69e8-e3bc-4d6d-b006-8cdb50510d59 to worker[182] qsize 0
2021-05-30 10:45:21,616 DEBUG awx.main.dispatch task e99e69e8-e3bc-4d6d-b006-8cdb50510d59 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:45:21,638 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:45:21,657 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:45:21.652921+00:00
2021-05-30 10:45:21,691 DEBUG awx.main.dispatch task e99e69e8-e3bc-4d6d-b006-8cdb50510d59 is finished
2021-05-30 10:45:31,591 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(04c41631-b534-4a18-826a-0567297cd8c0, queue=awx_private_queue)
[2021-05-30 10:45:31,591: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(04c41631-b534-4a18-826a-0567297cd8c0, queue=awx_private_queue)
2021-05-30 10:45:31,606 DEBUG awx.main.dispatch delivered 04c41631-b534-4a18-826a-0567297cd8c0 to worker[180] qsize 0
2021-05-30 10:45:31,612 DEBUG awx.main.dispatch task 04c41631-b534-4a18-826a-0567297cd8c0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:45:31,617 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:45:31,659 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:45:31,704 DEBUG awx.main.dispatch task 04c41631-b534-4a18-826a-0567297cd8c0 is finished
2021-05-30 10:45:51,594 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(985bf171-c60c-4d65-8a05-067f13078e43, queue=dev-awx-0)
[2021-05-30 10:45:51,594: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(985bf171-c60c-4d65-8a05-067f13078e43, queue=dev-awx-0)
2021-05-30 10:45:51,835 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c1547ccd-2a98-4658-bef8-712f26ffbf57, queue=dev-awx-0)
[2021-05-30 10:45:51,835: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c1547ccd-2a98-4658-bef8-712f26ffbf57, queue=dev-awx-0)
2021-05-30 10:45:51,850 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(10cb105a-4191-4d9e-bf9e-5e9f7cd3139b, queue=awx_private_queue)
[2021-05-30 10:45:51,850: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(10cb105a-4191-4d9e-bf9e-5e9f7cd3139b, queue=awx_private_queue)
2021-05-30 10:45:51,860 DEBUG awx.main.dispatch delivered 985bf171-c60c-4d65-8a05-067f13078e43 to worker[183] qsize 0
2021-05-30 10:45:51,865 DEBUG awx.main.dispatch delivered c1547ccd-2a98-4658-bef8-712f26ffbf57 to worker[180] qsize 0
2021-05-30 10:45:51,870 DEBUG awx.main.dispatch delivered 10cb105a-4191-4d9e-bf9e-5e9f7cd3139b to worker[182] qsize 0
2021-05-30 10:45:51,867 DEBUG awx.main.dispatch task 985bf171-c60c-4d65-8a05-067f13078e43 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 10:45:51,874 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:45:51,872 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6b1a70f9-e2ab-42f7-a701-514a6e2a4a1d, queue=awx_private_queue)
[2021-05-30 10:45:51,872: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6b1a70f9-e2ab-42f7-a701-514a6e2a4a1d, queue=awx_private_queue)
2021-05-30 10:45:51,877 DEBUG awx.main.dispatch task 10cb105a-4191-4d9e-bf9e-5e9f7cd3139b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:45:51,880 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:45:51,870 DEBUG awx.main.dispatch task c1547ccd-2a98-4658-bef8-712f26ffbf57 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 10:45:51,892 DEBUG awx.main.dispatch delivered 6b1a70f9-e2ab-42f7-a701-514a6e2a4a1d to worker[181] qsize 0
2021-05-30 10:45:51,899 DEBUG awx.main.dispatch task 6b1a70f9-e2ab-42f7-a701-514a6e2a4a1d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:45:51,907 DEBUG awx.main.dispatch task 985bf171-c60c-4d65-8a05-067f13078e43 is finished
2021-05-30 10:45:51,912 DEBUG awx.main.dispatch task c1547ccd-2a98-4658-bef8-712f26ffbf57 is finished
2021-05-30 10:45:51,916 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:45:51,921 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:45:51,921 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:45:21.639123+00:00
2021-05-30 10:45:51,943 DEBUG awx.main.dispatch task 6b1a70f9-e2ab-42f7-a701-514a6e2a4a1d is finished
2021-05-30 10:45:51,973 DEBUG awx.main.dispatch task 10cb105a-4191-4d9e-bf9e-5e9f7cd3139b is finished
RESULT 2
OKREADY
2021-05-30 10:46:11,860 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(119aa957-7527-47e9-bc90-2a5508b6aa10, queue=awx_private_queue)
[2021-05-30 10:46:11,860: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(119aa957-7527-47e9-bc90-2a5508b6aa10, queue=awx_private_queue)
2021-05-30 10:46:11,877 DEBUG awx.main.dispatch delivered 119aa957-7527-47e9-bc90-2a5508b6aa10 to worker[183] qsize 0
2021-05-30 10:46:11,879 DEBUG awx.main.dispatch task 119aa957-7527-47e9-bc90-2a5508b6aa10 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:46:11,881 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:46:11,896 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:46:11,930 DEBUG awx.main.dispatch task 119aa957-7527-47e9-bc90-2a5508b6aa10 is finished
2021-05-30 10:46:21,877 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(46545b42-7af9-4d76-9205-efa29015928a, queue=awx_private_queue)
[2021-05-30 10:46:21,877: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(46545b42-7af9-4d76-9205-efa29015928a, queue=awx_private_queue)
2021-05-30 10:46:21,903 DEBUG awx.main.dispatch delivered 46545b42-7af9-4d76-9205-efa29015928a to worker[180] qsize 0
2021-05-30 10:46:21,912 DEBUG awx.main.dispatch task 46545b42-7af9-4d76-9205-efa29015928a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:46:21,927 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:46:21,933 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:45:51.916825+00:00
2021-05-30 10:46:21,958 DEBUG awx.main.dispatch task 46545b42-7af9-4d76-9205-efa29015928a is finished
2021-05-30 10:46:31,860 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2c09070d-0a61-458a-94ae-e2d67e0d51cb, queue=awx_private_queue)
[2021-05-30 10:46:31,860: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2c09070d-0a61-458a-94ae-e2d67e0d51cb, queue=awx_private_queue)
2021-05-30 10:46:31,874 DEBUG awx.main.dispatch delivered 2c09070d-0a61-458a-94ae-e2d67e0d51cb to worker[180] qsize 0
2021-05-30 10:46:31,877 DEBUG awx.main.dispatch task 2c09070d-0a61-458a-94ae-e2d67e0d51cb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:46:31,879 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:46:31,897 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:46:31,923 DEBUG awx.main.dispatch task 2c09070d-0a61-458a-94ae-e2d67e0d51cb is finished
2021-05-30 10:46:51,600 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d7661d3f-32bb-4044-b0b4-d971940c3d5f, queue=dev-awx-0)
[2021-05-30 10:46:51,600: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d7661d3f-32bb-4044-b0b4-d971940c3d5f, queue=dev-awx-0)
2021-05-30 10:46:51,806 DEBUG awx.main.dispatch delivered d7661d3f-32bb-4044-b0b4-d971940c3d5f to worker[180] qsize 0
2021-05-30 10:46:51,808 DEBUG awx.main.dispatch task d7661d3f-32bb-4044-b0b4-d971940c3d5f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 10:46:51,810 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:46:51,831 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1ddab393-4da8-4cc9-8dbd-b6592e29d5ae, queue=dev-awx-0)
[2021-05-30 10:46:51,831: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1ddab393-4da8-4cc9-8dbd-b6592e29d5ae, queue=dev-awx-0)
2021-05-30 10:46:51,847 DEBUG awx.main.dispatch task d7661d3f-32bb-4044-b0b4-d971940c3d5f is finished
2021-05-30 10:46:51,851 DEBUG awx.main.dispatch delivered 1ddab393-4da8-4cc9-8dbd-b6592e29d5ae to worker[181] qsize 0
2021-05-30 10:46:51,854 DEBUG awx.main.dispatch task 1ddab393-4da8-4cc9-8dbd-b6592e29d5ae starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 10:46:51,859 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2770f57c-0040-4428-a1f0-4f8d9e90e546, queue=awx_private_queue)
[2021-05-30 10:46:51,859: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2770f57c-0040-4428-a1f0-4f8d9e90e546, queue=awx_private_queue)
2021-05-30 10:46:51,872 DEBUG awx.main.dispatch delivered 2770f57c-0040-4428-a1f0-4f8d9e90e546 to worker[180] qsize 0
2021-05-30 10:46:51,874 DEBUG awx.main.dispatch task 1ddab393-4da8-4cc9-8dbd-b6592e29d5ae is finished
2021-05-30 10:46:51,875 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dcdc5e02-b580-4536-8762-d0a9ba8a9b19, queue=awx_private_queue)
[2021-05-30 10:46:51,875: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dcdc5e02-b580-4536-8762-d0a9ba8a9b19, queue=awx_private_queue)
2021-05-30 10:46:51,875 DEBUG awx.main.dispatch task 2770f57c-0040-4428-a1f0-4f8d9e90e546 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:46:51,877 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:46:51,892 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:46:51,908 DEBUG awx.main.dispatch delivered dcdc5e02-b580-4536-8762-d0a9ba8a9b19 to worker[182] qsize 0
2021-05-30 10:46:51,912 DEBUG awx.main.dispatch task dcdc5e02-b580-4536-8762-d0a9ba8a9b19 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:46:51,928 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:46:51,927 DEBUG awx.main.dispatch task 2770f57c-0040-4428-a1f0-4f8d9e90e546 is finished
2021-05-30 10:46:51,933 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:46:21.927721+00:00
2021-05-30 10:46:51,946 DEBUG awx.main.dispatch task dcdc5e02-b580-4536-8762-d0a9ba8a9b19 is finished
RESULT 2
OKREADY
2021-05-30 10:47:11,872 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f57466ee-212d-4b36-9423-b1df705b901e, queue=awx_private_queue)
[2021-05-30 10:47:11,872: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f57466ee-212d-4b36-9423-b1df705b901e, queue=awx_private_queue)
2021-05-30 10:47:11,897 DEBUG awx.main.dispatch delivered f57466ee-212d-4b36-9423-b1df705b901e to worker[183] qsize 0
2021-05-30 10:47:11,907 DEBUG awx.main.dispatch task f57466ee-212d-4b36-9423-b1df705b901e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:47:11,911 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:47:11,931 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:47:11,957 DEBUG awx.main.dispatch task f57466ee-212d-4b36-9423-b1df705b901e is finished
2021-05-30 10:47:21,875 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(221511e2-5f84-4612-99d5-988d1f030fac, queue=awx_private_queue)
[2021-05-30 10:47:21,875: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(221511e2-5f84-4612-99d5-988d1f030fac, queue=awx_private_queue)
2021-05-30 10:47:21,898 DEBUG awx.main.dispatch delivered 221511e2-5f84-4612-99d5-988d1f030fac to worker[181] qsize 0
2021-05-30 10:47:21,901 DEBUG awx.main.dispatch task 221511e2-5f84-4612-99d5-988d1f030fac starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:47:21,915 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:47:21,919 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:46:51.928335+00:00
2021-05-30 10:47:21,932 DEBUG awx.main.dispatch task 221511e2-5f84-4612-99d5-988d1f030fac is finished
2021-05-30 10:47:31,870 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(50f209b4-ad4f-452b-867d-e7cf2394f36f, queue=awx_private_queue)
[2021-05-30 10:47:31,870: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(50f209b4-ad4f-452b-867d-e7cf2394f36f, queue=awx_private_queue)
2021-05-30 10:47:31,884 DEBUG awx.main.dispatch delivered 50f209b4-ad4f-452b-867d-e7cf2394f36f to worker[183] qsize 0
2021-05-30 10:47:31,887 DEBUG awx.main.dispatch task 50f209b4-ad4f-452b-867d-e7cf2394f36f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:47:31,889 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:47:31,908 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:47:31,937 DEBUG awx.main.dispatch task 50f209b4-ad4f-452b-867d-e7cf2394f36f is finished
2021-05-30 10:47:51,609 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8bda9dba-6181-43ff-8087-bc54707a783d, queue=dev-awx-0)
[2021-05-30 10:47:51,609: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8bda9dba-6181-43ff-8087-bc54707a783d, queue=dev-awx-0)
2021-05-30 10:47:51,803 DEBUG awx.main.dispatch delivered 8bda9dba-6181-43ff-8087-bc54707a783d to worker[182] qsize 0
2021-05-30 10:47:51,808 DEBUG awx.main.dispatch task 8bda9dba-6181-43ff-8087-bc54707a783d starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 10:47:51,811 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:47:51,831 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(8fafbfc5-55ad-4c83-9964-b941c76b3901, queue=dev-awx-0)
[2021-05-30 10:47:51,831: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(8fafbfc5-55ad-4c83-9964-b941c76b3901, queue=dev-awx-0)
2021-05-30 10:47:51,836 DEBUG awx.main.dispatch task 8bda9dba-6181-43ff-8087-bc54707a783d is finished
2021-05-30 10:47:51,856 DEBUG awx.main.dispatch delivered 8fafbfc5-55ad-4c83-9964-b941c76b3901 to worker[181] qsize 0
2021-05-30 10:47:51,858 DEBUG awx.main.dispatch task 8fafbfc5-55ad-4c83-9964-b941c76b3901 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 10:47:51,869 DEBUG awx.main.dispatch task 8fafbfc5-55ad-4c83-9964-b941c76b3901 is finished
2021-05-30 10:47:51,869 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7465fd72-febf-4778-bc04-acf3fda25c79, queue=awx_private_queue)
[2021-05-30 10:47:51,869: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7465fd72-febf-4778-bc04-acf3fda25c79, queue=awx_private_queue)
2021-05-30 10:47:51,882 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d4cbce8c-c304-4582-9fa1-fe7275f54663, queue=awx_private_queue)
[2021-05-30 10:47:51,882: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d4cbce8c-c304-4582-9fa1-fe7275f54663, queue=awx_private_queue)
2021-05-30 10:47:51,882 DEBUG awx.main.dispatch delivered 7465fd72-febf-4778-bc04-acf3fda25c79 to worker[182] qsize 0
2021-05-30 10:47:51,885 DEBUG awx.main.dispatch task 7465fd72-febf-4778-bc04-acf3fda25c79 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:47:51,886 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:47:51,895 DEBUG awx.main.dispatch delivered d4cbce8c-c304-4582-9fa1-fe7275f54663 to worker[181] qsize 0
2021-05-30 10:47:51,897 DEBUG awx.main.dispatch task d4cbce8c-c304-4582-9fa1-fe7275f54663 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:47:51,905 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:47:51,910 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:47:51,913 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:47:21.915430+00:00
2021-05-30 10:47:51,925 DEBUG awx.main.dispatch task d4cbce8c-c304-4582-9fa1-fe7275f54663 is finished
2021-05-30 10:47:51,930 DEBUG awx.main.dispatch task 7465fd72-febf-4778-bc04-acf3fda25c79 is finished
RESULT 2
OKREADY
2021-05-30 10:48:11,880 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(79cc9c18-d3a8-4bcc-9f1a-8b12aeaebff9, queue=awx_private_queue)
[2021-05-30 10:48:11,880: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(79cc9c18-d3a8-4bcc-9f1a-8b12aeaebff9, queue=awx_private_queue)
2021-05-30 10:48:11,904 DEBUG awx.main.dispatch delivered 79cc9c18-d3a8-4bcc-9f1a-8b12aeaebff9 to worker[180] qsize 0
2021-05-30 10:48:11,911 DEBUG awx.main.dispatch task 79cc9c18-d3a8-4bcc-9f1a-8b12aeaebff9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:48:11,916 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:48:11,950 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:48:11,987 DEBUG awx.main.dispatch task 79cc9c18-d3a8-4bcc-9f1a-8b12aeaebff9 is finished
2021-05-30 10:48:21,885 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(022b9fbb-03b1-45a8-80f9-d38685ca2f18, queue=awx_private_queue)
[2021-05-30 10:48:21,885: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(022b9fbb-03b1-45a8-80f9-d38685ca2f18, queue=awx_private_queue)
2021-05-30 10:48:21,908 DEBUG awx.main.dispatch delivered 022b9fbb-03b1-45a8-80f9-d38685ca2f18 to worker[181] qsize 0
2021-05-30 10:48:21,914 DEBUG awx.main.dispatch task 022b9fbb-03b1-45a8-80f9-d38685ca2f18 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:48:21,928 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:48:21,932 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:47:51.911244+00:00
2021-05-30 10:48:21,952 DEBUG awx.main.dispatch task 022b9fbb-03b1-45a8-80f9-d38685ca2f18 is finished
2021-05-30 10:48:31,878 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(62dc3dcc-8beb-4e55-a266-3432bbc49eb1, queue=awx_private_queue)
[2021-05-30 10:48:31,878: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(62dc3dcc-8beb-4e55-a266-3432bbc49eb1, queue=awx_private_queue)
2021-05-30 10:48:31,899 DEBUG awx.main.dispatch delivered 62dc3dcc-8beb-4e55-a266-3432bbc49eb1 to worker[181] qsize 0
2021-05-30 10:48:31,903 DEBUG awx.main.dispatch task 62dc3dcc-8beb-4e55-a266-3432bbc49eb1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:48:31,905 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:48:31,925 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:48:31,965 DEBUG awx.main.dispatch task 62dc3dcc-8beb-4e55-a266-3432bbc49eb1 is finished
2021-05-30 10:48:51,620 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(907c8c26-cd7a-4251-878f-d50cd028d058, queue=dev-awx-0)
[2021-05-30 10:48:51,620: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(907c8c26-cd7a-4251-878f-d50cd028d058, queue=dev-awx-0)
2021-05-30 10:48:51,875 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9eb83b51-62f1-40a7-97d5-cd7c3537c490, queue=dev-awx-0)
[2021-05-30 10:48:51,875: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9eb83b51-62f1-40a7-97d5-cd7c3537c490, queue=dev-awx-0)
2021-05-30 10:48:51,892 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6a501123-8952-4850-8ab2-4f128acc6543, queue=awx_private_queue)
[2021-05-30 10:48:51,892: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6a501123-8952-4850-8ab2-4f128acc6543, queue=awx_private_queue)
2021-05-30 10:48:51,892 DEBUG awx.main.dispatch delivered 907c8c26-cd7a-4251-878f-d50cd028d058 to worker[182] qsize 0
2021-05-30 10:48:51,895 DEBUG awx.main.dispatch delivered 9eb83b51-62f1-40a7-97d5-cd7c3537c490 to worker[181] qsize 0
2021-05-30 10:48:51,897 DEBUG awx.main.dispatch task 907c8c26-cd7a-4251-878f-d50cd028d058 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 10:48:51,900 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:48:51,898 DEBUG awx.main.dispatch task 9eb83b51-62f1-40a7-97d5-cd7c3537c490 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 10:48:51,914 DEBUG awx.main.dispatch delivered 6a501123-8952-4850-8ab2-4f128acc6543 to worker[180] qsize 0
2021-05-30 10:48:51,915 DEBUG awx.main.dispatch task 9eb83b51-62f1-40a7-97d5-cd7c3537c490 is finished
2021-05-30 10:48:51,916 DEBUG awx.main.dispatch task 6a501123-8952-4850-8ab2-4f128acc6543 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:48:51,917 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:48:51,915 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(218b0dd1-4354-47ac-aa0a-fb5f2d723a26, queue=awx_private_queue)
[2021-05-30 10:48:51,915: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(218b0dd1-4354-47ac-aa0a-fb5f2d723a26, queue=awx_private_queue)
2021-05-30 10:48:51,928 DEBUG awx.main.dispatch task 907c8c26-cd7a-4251-878f-d50cd028d058 is finished
2021-05-30 10:48:51,934 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:48:51,935 DEBUG awx.main.dispatch delivered 218b0dd1-4354-47ac-aa0a-fb5f2d723a26 to worker[181] qsize 0
2021-05-30 10:48:51,936 DEBUG awx.main.dispatch task 218b0dd1-4354-47ac-aa0a-fb5f2d723a26 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:48:51,945 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:48:51,947 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:48:21.928249+00:00
2021-05-30 10:48:51,957 DEBUG awx.main.dispatch task 218b0dd1-4354-47ac-aa0a-fb5f2d723a26 is finished
2021-05-30 10:48:51,965 DEBUG awx.main.dispatch task 6a501123-8952-4850-8ab2-4f128acc6543 is finished
RESULT 2
OKREADY
2021-05-30 10:49:11,892 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d4de1d6e-bd5d-4e29-b86a-a00166484915, queue=awx_private_queue)
[2021-05-30 10:49:11,892: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d4de1d6e-bd5d-4e29-b86a-a00166484915, queue=awx_private_queue)
2021-05-30 10:49:11,908 DEBUG awx.main.dispatch delivered d4de1d6e-bd5d-4e29-b86a-a00166484915 to worker[183] qsize 0
2021-05-30 10:49:11,913 DEBUG awx.main.dispatch task d4de1d6e-bd5d-4e29-b86a-a00166484915 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:49:11,917 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:49:11,936 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:49:11,963 DEBUG awx.main.dispatch task d4de1d6e-bd5d-4e29-b86a-a00166484915 is finished
2021-05-30 10:49:21,916 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(30116d0d-4b92-4797-afce-66d3b1ef636c, queue=awx_private_queue)
[2021-05-30 10:49:21,916: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(30116d0d-4b92-4797-afce-66d3b1ef636c, queue=awx_private_queue)
2021-05-30 10:49:21,938 DEBUG awx.main.dispatch delivered 30116d0d-4b92-4797-afce-66d3b1ef636c to worker[183] qsize 0
2021-05-30 10:49:21,941 DEBUG awx.main.dispatch task 30116d0d-4b92-4797-afce-66d3b1ef636c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:49:21,956 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:49:21,961 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:48:51.945136+00:00
2021-05-30 10:49:21,985 DEBUG awx.main.dispatch task 30116d0d-4b92-4797-afce-66d3b1ef636c is finished
2021-05-30 10:49:31,893 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3a63c17e-b1b5-4927-b9a1-80393fa303df, queue=awx_private_queue)
[2021-05-30 10:49:31,893: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3a63c17e-b1b5-4927-b9a1-80393fa303df, queue=awx_private_queue)
2021-05-30 10:49:31,912 DEBUG awx.main.dispatch delivered 3a63c17e-b1b5-4927-b9a1-80393fa303df to worker[182] qsize 0
2021-05-30 10:49:31,916 DEBUG awx.main.dispatch task 3a63c17e-b1b5-4927-b9a1-80393fa303df starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:49:31,919 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:49:31,938 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:49:31,969 DEBUG awx.main.dispatch task 3a63c17e-b1b5-4927-b9a1-80393fa303df is finished
2021-05-30 10:49:51,634 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(0b9b4bdf-6326-468d-8bac-884bfc95eb9b, queue=dev-awx-0)
[2021-05-30 10:49:51,634: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(0b9b4bdf-6326-468d-8bac-884bfc95eb9b, queue=dev-awx-0)
2021-05-30 10:49:51,875 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(482f51f4-b48a-4ad1-bcec-891414aa8446, queue=dev-awx-0)
[2021-05-30 10:49:51,875: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(482f51f4-b48a-4ad1-bcec-891414aa8446, queue=dev-awx-0)
2021-05-30 10:49:51,892 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(61d41bc8-0ffd-4826-a106-f9129677ad1c, queue=awx_private_queue)
[2021-05-30 10:49:51,892: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(61d41bc8-0ffd-4826-a106-f9129677ad1c, queue=awx_private_queue)
2021-05-30 10:49:51,894 DEBUG awx.main.dispatch delivered 0b9b4bdf-6326-468d-8bac-884bfc95eb9b to worker[182] qsize 0
2021-05-30 10:49:51,900 DEBUG awx.main.dispatch task 0b9b4bdf-6326-468d-8bac-884bfc95eb9b starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 10:49:51,904 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:49:51,903 DEBUG awx.main.dispatch delivered 482f51f4-b48a-4ad1-bcec-891414aa8446 to worker[180] qsize 0
2021-05-30 10:49:51,909 DEBUG awx.main.dispatch delivered 61d41bc8-0ffd-4826-a106-f9129677ad1c to worker[181] qsize 0
2021-05-30 10:49:51,908 DEBUG awx.main.dispatch task 482f51f4-b48a-4ad1-bcec-891414aa8446 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 10:49:51,915 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0d78a910-7336-4779-af22-aca8966a2dfc, queue=awx_private_queue)
[2021-05-30 10:49:51,915: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0d78a910-7336-4779-af22-aca8966a2dfc, queue=awx_private_queue)
2021-05-30 10:49:51,915 DEBUG awx.main.dispatch task 61d41bc8-0ffd-4826-a106-f9129677ad1c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:49:51,922 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:49:51,927 DEBUG awx.main.dispatch task 482f51f4-b48a-4ad1-bcec-891414aa8446 is finished
2021-05-30 10:49:51,931 DEBUG awx.main.dispatch task 0b9b4bdf-6326-468d-8bac-884bfc95eb9b is finished
2021-05-30 10:49:51,933 DEBUG awx.main.dispatch delivered 0d78a910-7336-4779-af22-aca8966a2dfc to worker[183] qsize 0
2021-05-30 10:49:51,936 DEBUG awx.main.dispatch task 0d78a910-7336-4779-af22-aca8966a2dfc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:49:51,947 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:49:51,949 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:49:51,950 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:49:21.956646+00:00
2021-05-30 10:49:51,962 DEBUG awx.main.dispatch task 0d78a910-7336-4779-af22-aca8966a2dfc is finished
2021-05-30 10:49:51,985 DEBUG awx.main.dispatch task 61d41bc8-0ffd-4826-a106-f9129677ad1c is finished
RESULT 2
OKREADY
2021-05-30 10:50:11,900 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fdb2bae1-6b25-4fd2-bdc1-4a7aef73ce4f, queue=awx_private_queue)
[2021-05-30 10:50:11,900: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fdb2bae1-6b25-4fd2-bdc1-4a7aef73ce4f, queue=awx_private_queue)
2021-05-30 10:50:11,918 DEBUG awx.main.dispatch delivered fdb2bae1-6b25-4fd2-bdc1-4a7aef73ce4f to worker[183] qsize 0
2021-05-30 10:50:11,921 DEBUG awx.main.dispatch task fdb2bae1-6b25-4fd2-bdc1-4a7aef73ce4f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:50:11,923 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:50:11,945 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:50:11,976 DEBUG awx.main.dispatch task fdb2bae1-6b25-4fd2-bdc1-4a7aef73ce4f is finished
2021-05-30 10:50:21,915 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(64b86fff-8c98-48c3-a207-48e9fb11d20c, queue=awx_private_queue)
[2021-05-30 10:50:21,915: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(64b86fff-8c98-48c3-a207-48e9fb11d20c, queue=awx_private_queue)
2021-05-30 10:50:21,930 DEBUG awx.main.dispatch delivered 64b86fff-8c98-48c3-a207-48e9fb11d20c to worker[182] qsize 0
2021-05-30 10:50:21,932 DEBUG awx.main.dispatch task 64b86fff-8c98-48c3-a207-48e9fb11d20c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:50:21,943 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:50:21,946 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:49:51.947443+00:00
2021-05-30 10:50:21,962 DEBUG awx.main.dispatch task 64b86fff-8c98-48c3-a207-48e9fb11d20c is finished
2021-05-30 10:50:31,900 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6c40f070-3b3c-434c-975d-9ca33c8d9d89, queue=awx_private_queue)
[2021-05-30 10:50:31,900: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6c40f070-3b3c-434c-975d-9ca33c8d9d89, queue=awx_private_queue)
2021-05-30 10:50:31,924 DEBUG awx.main.dispatch delivered 6c40f070-3b3c-434c-975d-9ca33c8d9d89 to worker[183] qsize 0
2021-05-30 10:50:31,928 DEBUG awx.main.dispatch task 6c40f070-3b3c-434c-975d-9ca33c8d9d89 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:50:31,929 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:50:31,952 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:50:31,991 DEBUG awx.main.dispatch task 6c40f070-3b3c-434c-975d-9ca33c8d9d89 is finished
2021-05-30 10:50:51,637 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(86655f15-6a49-42dd-b284-56a6a5435ce0, queue=dev-awx-0)
[2021-05-30 10:50:51,637: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(86655f15-6a49-42dd-b284-56a6a5435ce0, queue=dev-awx-0)
2021-05-30 10:50:51,874 DEBUG awx.main.dispatch delivered 86655f15-6a49-42dd-b284-56a6a5435ce0 to worker[180] qsize 0
2021-05-30 10:50:51,875 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(76c3824d-a687-492a-97cd-feeb0422c592, queue=dev-awx-0)
[2021-05-30 10:50:51,875: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(76c3824d-a687-492a-97cd-feeb0422c592, queue=dev-awx-0)
2021-05-30 10:50:51,876 DEBUG awx.main.dispatch task 86655f15-6a49-42dd-b284-56a6a5435ce0 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 10:50:51,878 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:50:51,887 DEBUG awx.main.dispatch delivered 76c3824d-a687-492a-97cd-feeb0422c592 to worker[183] qsize 0
2021-05-30 10:50:51,889 DEBUG awx.main.dispatch task 76c3824d-a687-492a-97cd-feeb0422c592 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 10:50:51,898 DEBUG awx.main.dispatch task 86655f15-6a49-42dd-b284-56a6a5435ce0 is finished
2021-05-30 10:50:51,899 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(678195dd-c79c-480a-aff5-b5326bd4068f, queue=awx_private_queue)
[2021-05-30 10:50:51,899: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(678195dd-c79c-480a-aff5-b5326bd4068f, queue=awx_private_queue)
2021-05-30 10:50:51,905 DEBUG awx.main.dispatch task 76c3824d-a687-492a-97cd-feeb0422c592 is finished
2021-05-30 10:50:51,911 DEBUG awx.main.dispatch delivered 678195dd-c79c-480a-aff5-b5326bd4068f to worker[181] qsize 0
2021-05-30 10:50:51,914 DEBUG awx.main.dispatch task 678195dd-c79c-480a-aff5-b5326bd4068f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:50:51,916 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:50:51,915 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aeee9489-702e-4e1e-af5f-9fba09457d26, queue=awx_private_queue)
[2021-05-30 10:50:51,915: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aeee9489-702e-4e1e-af5f-9fba09457d26, queue=awx_private_queue)
2021-05-30 10:50:51,928 DEBUG awx.main.dispatch delivered aeee9489-702e-4e1e-af5f-9fba09457d26 to worker[183] qsize 0
2021-05-30 10:50:51,929 DEBUG awx.main.dispatch task aeee9489-702e-4e1e-af5f-9fba09457d26 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:50:51,933 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:50:51,941 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:50:51,945 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:50:21.944094+00:00
2021-05-30 10:50:51,962 DEBUG awx.main.dispatch task aeee9489-702e-4e1e-af5f-9fba09457d26 is finished
2021-05-30 10:50:51,959 DEBUG awx.main.dispatch task 678195dd-c79c-480a-aff5-b5326bd4068f is finished
RESULT 2
OKREADY
2021-05-30 10:51:11,908 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8af75766-a7e7-44ce-88c2-b9df705d06ed, queue=awx_private_queue)
[2021-05-30 10:51:11,908: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8af75766-a7e7-44ce-88c2-b9df705d06ed, queue=awx_private_queue)
2021-05-30 10:51:11,937 DEBUG awx.main.dispatch delivered 8af75766-a7e7-44ce-88c2-b9df705d06ed to worker[183] qsize 0
2021-05-30 10:51:11,945 DEBUG awx.main.dispatch task 8af75766-a7e7-44ce-88c2-b9df705d06ed starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:51:11,952 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:51:11,975 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:51:12,010 DEBUG awx.main.dispatch task 8af75766-a7e7-44ce-88c2-b9df705d06ed is finished
2021-05-30 10:51:21,917 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2229bcc7-e5ba-4758-98ef-fb954b00e22b, queue=awx_private_queue)
[2021-05-30 10:51:21,917: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2229bcc7-e5ba-4758-98ef-fb954b00e22b, queue=awx_private_queue)
2021-05-30 10:51:21,940 DEBUG awx.main.dispatch delivered 2229bcc7-e5ba-4758-98ef-fb954b00e22b to worker[183] qsize 0
2021-05-30 10:51:21,942 DEBUG awx.main.dispatch task 2229bcc7-e5ba-4758-98ef-fb954b00e22b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:51:21,953 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:51:21,959 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:50:51.942081+00:00
2021-05-30 10:51:21,978 DEBUG awx.main.dispatch task 2229bcc7-e5ba-4758-98ef-fb954b00e22b is finished
2021-05-30 10:51:31,904 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(81044934-bf26-405d-91bd-b7663a74e4ce, queue=awx_private_queue)
[2021-05-30 10:51:31,904: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(81044934-bf26-405d-91bd-b7663a74e4ce, queue=awx_private_queue)
2021-05-30 10:51:31,929 DEBUG awx.main.dispatch delivered 81044934-bf26-405d-91bd-b7663a74e4ce to worker[180] qsize 0
2021-05-30 10:51:31,935 DEBUG awx.main.dispatch task 81044934-bf26-405d-91bd-b7663a74e4ce starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:51:31,941 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:51:31,967 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:51:32,001 DEBUG awx.main.dispatch task 81044934-bf26-405d-91bd-b7663a74e4ce is finished
2021-05-30 10:51:51,637 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(15ee9f34-8dae-4625-8f1f-2b85aef9d152, queue=dev-awx-0)
[2021-05-30 10:51:51,637: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(15ee9f34-8dae-4625-8f1f-2b85aef9d152, queue=dev-awx-0)
2021-05-30 10:51:51,885 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(72b5b194-b21e-4822-94d2-6a8ba3c4d75d, queue=dev-awx-0)
[2021-05-30 10:51:51,885: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(72b5b194-b21e-4822-94d2-6a8ba3c4d75d, queue=dev-awx-0)
2021-05-30 10:51:51,903 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9d749841-b31f-494b-9a2c-f8614e7b5408, queue=awx_private_queue)
[2021-05-30 10:51:51,903: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9d749841-b31f-494b-9a2c-f8614e7b5408, queue=awx_private_queue)
2021-05-30 10:51:51,905 DEBUG awx.main.dispatch delivered 15ee9f34-8dae-4625-8f1f-2b85aef9d152 to worker[180] qsize 0
2021-05-30 10:51:51,908 DEBUG awx.main.dispatch task 15ee9f34-8dae-4625-8f1f-2b85aef9d152 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 10:51:51,910 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:51:51,908 DEBUG awx.main.dispatch delivered 72b5b194-b21e-4822-94d2-6a8ba3c4d75d to worker[181] qsize 0
2021-05-30 10:51:51,913 DEBUG awx.main.dispatch task 72b5b194-b21e-4822-94d2-6a8ba3c4d75d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 10:51:51,922 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1c9eb08e-dbf8-4888-bed0-b80c2cd4a453, queue=awx_private_queue)
[2021-05-30 10:51:51,922: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1c9eb08e-dbf8-4888-bed0-b80c2cd4a453, queue=awx_private_queue)
2021-05-30 10:51:51,921 DEBUG awx.main.dispatch delivered 9d749841-b31f-494b-9a2c-f8614e7b5408 to worker[183] qsize 0
2021-05-30 10:51:51,926 DEBUG awx.main.dispatch task 9d749841-b31f-494b-9a2c-f8614e7b5408 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:51:51,928 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:51:51,936 DEBUG awx.main.dispatch delivered 1c9eb08e-dbf8-4888-bed0-b80c2cd4a453 to worker[182] qsize 0
2021-05-30 10:51:51,941 DEBUG awx.main.dispatch task 15ee9f34-8dae-4625-8f1f-2b85aef9d152 is finished
2021-05-30 10:51:51,942 DEBUG awx.main.dispatch task 1c9eb08e-dbf8-4888-bed0-b80c2cd4a453 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:51:51,958 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:51:51,963 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:51:21.954212+00:00
2021-05-30 10:51:51,967 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:51:51,966 DEBUG awx.main.dispatch task 72b5b194-b21e-4822-94d2-6a8ba3c4d75d is finished
2021-05-30 10:51:51,977 DEBUG awx.main.dispatch task 1c9eb08e-dbf8-4888-bed0-b80c2cd4a453 is finished
2021-05-30 10:51:52,002 DEBUG awx.main.dispatch task 9d749841-b31f-494b-9a2c-f8614e7b5408 is finished
RESULT 2
OKREADY
2021-05-30 10:52:11,914 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(896eec1a-6fc7-4bff-8dd7-4c719ed61f4e, queue=awx_private_queue)
[2021-05-30 10:52:11,914: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(896eec1a-6fc7-4bff-8dd7-4c719ed61f4e, queue=awx_private_queue)
2021-05-30 10:52:11,937 DEBUG awx.main.dispatch delivered 896eec1a-6fc7-4bff-8dd7-4c719ed61f4e to worker[183] qsize 0
2021-05-30 10:52:11,940 DEBUG awx.main.dispatch task 896eec1a-6fc7-4bff-8dd7-4c719ed61f4e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:52:11,941 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:52:11,961 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:52:11,999 DEBUG awx.main.dispatch task 896eec1a-6fc7-4bff-8dd7-4c719ed61f4e is finished
2021-05-30 10:52:21,922 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(81fd7aed-ce50-4716-bed7-10b725fb388f, queue=awx_private_queue)
[2021-05-30 10:52:21,922: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(81fd7aed-ce50-4716-bed7-10b725fb388f, queue=awx_private_queue)
2021-05-30 10:52:21,937 DEBUG awx.main.dispatch delivered 81fd7aed-ce50-4716-bed7-10b725fb388f to worker[181] qsize 0
2021-05-30 10:52:21,940 DEBUG awx.main.dispatch task 81fd7aed-ce50-4716-bed7-10b725fb388f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:52:21,976 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:52:21,996 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:51:51.959107+00:00
2021-05-30 10:52:22,038 DEBUG awx.main.dispatch task 81fd7aed-ce50-4716-bed7-10b725fb388f is finished
2021-05-30 10:52:31,914 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8812bc84-be14-442a-8f98-2954367070fa, queue=awx_private_queue)
[2021-05-30 10:52:31,914: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8812bc84-be14-442a-8f98-2954367070fa, queue=awx_private_queue)
2021-05-30 10:52:31,929 DEBUG awx.main.dispatch delivered 8812bc84-be14-442a-8f98-2954367070fa to worker[181] qsize 0
2021-05-30 10:52:31,931 DEBUG awx.main.dispatch task 8812bc84-be14-442a-8f98-2954367070fa starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:52:31,932 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:52:31,954 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:52:31,985 DEBUG awx.main.dispatch task 8812bc84-be14-442a-8f98-2954367070fa is finished
2021-05-30 10:52:51,657 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6a461667-d0cb-45b6-982c-414524bf302b, queue=dev-awx-0)
[2021-05-30 10:52:51,657: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6a461667-d0cb-45b6-982c-414524bf302b, queue=dev-awx-0)
2021-05-30 10:52:51,720 DEBUG awx.main.dispatch delivered 6a461667-d0cb-45b6-982c-414524bf302b to worker[181] qsize 0
2021-05-30 10:52:51,731 DEBUG awx.main.dispatch task 6a461667-d0cb-45b6-982c-414524bf302b starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 10:52:51,734 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:52:51,885 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(83f9bb09-86b0-4052-91c3-d5e29ba13e58, queue=dev-awx-0)
[2021-05-30 10:52:51,885: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(83f9bb09-86b0-4052-91c3-d5e29ba13e58, queue=dev-awx-0)
2021-05-30 10:52:51,901 DEBUG awx.main.dispatch delivered 83f9bb09-86b0-4052-91c3-d5e29ba13e58 to worker[180] qsize 0
2021-05-30 10:52:51,913 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b2893d32-50ad-4bba-a98c-8e5f3ef116c9, queue=awx_private_queue)
[2021-05-30 10:52:51,913: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b2893d32-50ad-4bba-a98c-8e5f3ef116c9, queue=awx_private_queue)
2021-05-30 10:52:51,930 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7be1c997-6f36-4fb9-bab6-baaf3cf8776b, queue=awx_private_queue)
[2021-05-30 10:52:51,930: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7be1c997-6f36-4fb9-bab6-baaf3cf8776b, queue=awx_private_queue)
2021-05-30 10:52:51,928 DEBUG awx.main.dispatch delivered b2893d32-50ad-4bba-a98c-8e5f3ef116c9 to worker[183] qsize 0
2021-05-30 10:52:52,115 DEBUG awx.main.dispatch delivered 7be1c997-6f36-4fb9-bab6-baaf3cf8776b to worker[182] qsize 0
2021-05-30 10:52:52,118 DEBUG awx.main.dispatch task 7be1c997-6f36-4fb9-bab6-baaf3cf8776b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:52:52,119 DEBUG awx.main.dispatch task b2893d32-50ad-4bba-a98c-8e5f3ef116c9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:52:52,123 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:52:51,905 DEBUG awx.main.dispatch task 83f9bb09-86b0-4052-91c3-d5e29ba13e58 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 10:52:52,144 DEBUG awx.main.dispatch task 83f9bb09-86b0-4052-91c3-d5e29ba13e58 is finished
2021-05-30 10:52:51,969 DEBUG awx.main.dispatch task 6a461667-d0cb-45b6-982c-414524bf302b is finished
2021-05-30 10:52:52,282 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:52:52,287 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:52:21.976843+00:00
2021-05-30 10:52:52,293 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:52:52,312 DEBUG awx.main.dispatch task 7be1c997-6f36-4fb9-bab6-baaf3cf8776b is finished
2021-05-30 10:52:52,326 DEBUG awx.main.dispatch task b2893d32-50ad-4bba-a98c-8e5f3ef116c9 is finished
RESULT 2
OKREADY
2021-05-30 10:53:11,925 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2d112e8f-41af-410e-887a-fb9e8182d58f, queue=awx_private_queue)
[2021-05-30 10:53:11,925: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2d112e8f-41af-410e-887a-fb9e8182d58f, queue=awx_private_queue)
2021-05-30 10:53:11,950 DEBUG awx.main.dispatch delivered 2d112e8f-41af-410e-887a-fb9e8182d58f to worker[180] qsize 0
2021-05-30 10:53:11,953 DEBUG awx.main.dispatch task 2d112e8f-41af-410e-887a-fb9e8182d58f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:53:11,956 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:53:11,988 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:53:12,027 DEBUG awx.main.dispatch task 2d112e8f-41af-410e-887a-fb9e8182d58f is finished
2021-05-30 10:53:21,930 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(11150bad-2a9d-43b1-8ee7-79781c250745, queue=awx_private_queue)
[2021-05-30 10:53:21,930: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(11150bad-2a9d-43b1-8ee7-79781c250745, queue=awx_private_queue)
2021-05-30 10:53:21,945 DEBUG awx.main.dispatch delivered 11150bad-2a9d-43b1-8ee7-79781c250745 to worker[180] qsize 0
2021-05-30 10:53:21,947 DEBUG awx.main.dispatch task 11150bad-2a9d-43b1-8ee7-79781c250745 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:53:21,956 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:53:21,959 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:52:52.282646+00:00
2021-05-30 10:53:22,007 DEBUG awx.main.dispatch task 11150bad-2a9d-43b1-8ee7-79781c250745 is finished
2021-05-30 10:53:31,926 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6b143a99-cb98-45c5-b2df-1158e9c4a79e, queue=awx_private_queue)
[2021-05-30 10:53:31,926: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6b143a99-cb98-45c5-b2df-1158e9c4a79e, queue=awx_private_queue)
2021-05-30 10:53:31,952 DEBUG awx.main.dispatch delivered 6b143a99-cb98-45c5-b2df-1158e9c4a79e to worker[180] qsize 0
2021-05-30 10:53:31,955 DEBUG awx.main.dispatch task 6b143a99-cb98-45c5-b2df-1158e9c4a79e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:53:31,957 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:53:31,980 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:53:32,014 DEBUG awx.main.dispatch task 6b143a99-cb98-45c5-b2df-1158e9c4a79e is finished
2021-05-30 10:53:51,658 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(2de18a4c-542f-4eed-aeed-e39990271df4, queue=dev-awx-0)
[2021-05-30 10:53:51,658: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(2de18a4c-542f-4eed-aeed-e39990271df4, queue=dev-awx-0)
2021-05-30 10:53:51,729 DEBUG awx.main.dispatch delivered 2de18a4c-542f-4eed-aeed-e39990271df4 to worker[182] qsize 0
2021-05-30 10:53:51,735 DEBUG awx.main.dispatch task 2de18a4c-542f-4eed-aeed-e39990271df4 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 10:53:51,770 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 10:53:51,794 DEBUG awx.main.dispatch task 2de18a4c-542f-4eed-aeed-e39990271df4 is finished
2021-05-30 10:53:51,886 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(33d54b65-eea7-491b-8121-87fc7d3c2cd9, queue=dev-awx-0)
[2021-05-30 10:53:51,886: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(33d54b65-eea7-491b-8121-87fc7d3c2cd9, queue=dev-awx-0)
2021-05-30 10:53:51,911 DEBUG awx.main.dispatch delivered 33d54b65-eea7-491b-8121-87fc7d3c2cd9 to worker[181] qsize 0
2021-05-30 10:53:51,916 DEBUG awx.main.dispatch task 33d54b65-eea7-491b-8121-87fc7d3c2cd9 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 10:53:51,930 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(08e354f1-6f9d-434f-a277-7716897328f1, queue=awx_private_queue)
[2021-05-30 10:53:51,930: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(08e354f1-6f9d-434f-a277-7716897328f1, queue=awx_private_queue)
2021-05-30 10:53:51,948 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(77b25ebb-aec4-430d-a749-850d2e5b29e4, queue=awx_private_queue)
[2021-05-30 10:53:51,948: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(77b25ebb-aec4-430d-a749-850d2e5b29e4, queue=awx_private_queue)
2021-05-30 10:53:51,965 DEBUG awx.main.dispatch task 33d54b65-eea7-491b-8121-87fc7d3c2cd9 is finished
2021-05-30 10:53:51,947 DEBUG awx.main.dispatch delivered 08e354f1-6f9d-434f-a277-7716897328f1 to worker[180] qsize 0
2021-05-30 10:53:52,155 DEBUG awx.main.dispatch delivered 77b25ebb-aec4-430d-a749-850d2e5b29e4 to worker[183] qsize 0
2021-05-30 10:53:52,156 DEBUG awx.main.dispatch task 08e354f1-6f9d-434f-a277-7716897328f1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:53:52,159 DEBUG awx.main.dispatch task 77b25ebb-aec4-430d-a749-850d2e5b29e4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:53:52,163 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:53:52,187 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:53:52,191 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:53:21.956898+00:00
2021-05-30 10:53:52,199 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:53:52,235 DEBUG awx.main.dispatch task 77b25ebb-aec4-430d-a749-850d2e5b29e4 is finished
2021-05-30 10:53:52,268 DEBUG awx.main.dispatch task 08e354f1-6f9d-434f-a277-7716897328f1 is finished
RESULT 2
OKREADY
2021-05-30 10:54:11,955 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(47921b45-a771-415d-84b3-1de5b18bdb39, queue=awx_private_queue)
[2021-05-30 10:54:11,955: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(47921b45-a771-415d-84b3-1de5b18bdb39, queue=awx_private_queue)
2021-05-30 10:54:11,977 DEBUG awx.main.dispatch delivered 47921b45-a771-415d-84b3-1de5b18bdb39 to worker[182] qsize 0
2021-05-30 10:54:11,982 DEBUG awx.main.dispatch task 47921b45-a771-415d-84b3-1de5b18bdb39 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:54:11,985 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:54:12,009 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:54:12,055 DEBUG awx.main.dispatch task 47921b45-a771-415d-84b3-1de5b18bdb39 is finished
2021-05-30 10:54:21,930 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a36b7808-1fe3-4224-b0af-7d55d19a808c, queue=awx_private_queue)
[2021-05-30 10:54:21,930: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a36b7808-1fe3-4224-b0af-7d55d19a808c, queue=awx_private_queue)
2021-05-30 10:54:21,953 DEBUG awx.main.dispatch delivered a36b7808-1fe3-4224-b0af-7d55d19a808c to worker[183] qsize 0
2021-05-30 10:54:21,957 DEBUG awx.main.dispatch task a36b7808-1fe3-4224-b0af-7d55d19a808c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 10:54:29,432 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 10:54:29,463 DEBUG awx.main.tasks Last scheduler run was: 2021-05-30 10:53:52.187171+00:00
2021-05-30 10:54:29,652 DEBUG awx.main.dispatch task a36b7808-1fe3-4224-b0af-7d55d19a808c is finished
2021-05-30 10:54:31,955 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0c9361b1-5671-4a78-a3c0-fef6a9c2f77d, queue=awx_private_queue)
[2021-05-30 10:54:31,955: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0c9361b1-5671-4a78-a3c0-fef6a9c2f77d, queue=awx_private_queue)
2021-05-30 10:54:31,987 DEBUG awx.main.dispatch delivered 0c9361b1-5671-4a78-a3c0-fef6a9c2f77d to worker[182] qsize 0
2021-05-30 10:54:31,990 DEBUG awx.main.dispatch task 0c9361b1-5671-4a78-a3c0-fef6a9c2f77d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 10:54:31,991 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 10:54:33,202 DEBUG awx.main.scheduler Starting Scheduler
2021-05-30 10:54:33,354 DEBUG awx.main.dispatch task 0c9361b1-5671-4a78-a3c0-fef6a9c2f77d is finished
2021-05-30 10:54:51,588 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(eda1b0dc-9b80-4b5b-84cb-d44d11d68ab3, queue=dev-awx-0)
[2021-05-30 10:54:51,588: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(eda1b0dc-9b80-4b5b-84cb-d44d11d68ab3, queue=dev-awx-0)
2021-05-30 10:54:51,608 DEBUG awx.main.dispatch delivered eda1b0dc-9b80-4b5b-84cb-d44d11d68ab3 to worker[182] qsize 0
2021-05-30 10:54:51,611 DEBUG awx.main.dispatch task eda1b0dc-9b80-4b5b-84cb-d44d11d68ab3 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-30 10:54:51,612 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-30 10:54:51,653 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(03a10ff4-9608-4646-ad13-27005cbc92e5, queue=dev-awx-0)
[2021-05-30 10:54:51,653: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(03a10ff4-9608-4646-ad13-27005cbc92e5, queue=dev-awx-0)
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
2021-05-30 10:54:51,885 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(64e567fe-899e-492f-b8db-56ca1b051452, queue=dev-awx-0)
[2021-05-30 10:54:51,885: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(64e567fe-899e-492f-b8db-56ca1b051452, queue=dev-awx-0)
2021-05-30 11:01:36,292 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7ccd1617-72c3-4165-9fdd-26eebcb5cc8d, queue=awx_private_queue)
[2021-05-30 11:01:36,292: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7ccd1617-72c3-4165-9fdd-26eebcb5cc8d, queue=awx_private_queue)
2021-05-30 11:01:36,309 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(00781c13-dd79-43e1-9796-5357ca6a30da, queue=awx_private_queue)
[2021-05-30 11:01:36,309: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(00781c13-dd79-43e1-9796-5357ca6a30da, queue=awx_private_queue)
2021-05-30 11:01:36,325 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d582adf4-412b-4d58-9b01-5bc0fef2debb, queue=dev-awx-0)
[2021-05-30 11:01:36,325: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d582adf4-412b-4d58-9b01-5bc0fef2debb, queue=dev-awx-0)
2021-05-30 11:01:35,371 DEBUG awx.main.dispatch delivered 03a10ff4-9608-4646-ad13-27005cbc92e5 to worker[180] qsize 0
2021-05-30 11:01:36,446 DEBUG awx.main.dispatch delivered 64e567fe-899e-492f-b8db-56ca1b051452 to worker[181] qsize 0
2021-05-30 11:01:36,448 DEBUG awx.main.dispatch delivered 7ccd1617-72c3-4165-9fdd-26eebcb5cc8d to worker[183] qsize 0
2021-05-30 11:01:36,447 DEBUG awx.main.dispatch task 03a10ff4-9608-4646-ad13-27005cbc92e5 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-05-30 11:01:36,453 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-05-30 11:01:36,450 DEBUG awx.main.dispatch task 64e567fe-899e-492f-b8db-56ca1b051452 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:01:36,451 DEBUG awx.main.dispatch task 7ccd1617-72c3-4165-9fdd-26eebcb5cc8d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:01:36,470 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:01:36,469 WARNING awx.main.dispatch scaling up worker pid:192
2021-05-30 11:01:36,468 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-30 11:01:36,476 DEBUG awx.main.dispatch task 03a10ff4-9608-4646-ad13-27005cbc92e5 is finished
2021-05-30 11:01:36,477 DEBUG awx.main.dispatch delivered 00781c13-dd79-43e1-9796-5357ca6a30da to worker[192] qsize 0
2021-05-30 11:01:36,472 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_6" does not exist
2021-05-30 11:01:36,479 DEBUG awx.main.dispatch task 64e567fe-899e-492f-b8db-56ca1b051452 is finished
2021-05-30 11:01:36,483 WARNING awx.main.dispatch scaling down worker pid:180
2021-05-30 11:01:36,485 WARNING awx.main.dispatch worker exiting gracefully pid:180
2021-05-30 11:01:36,478 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:01:36,487 DEBUG awx.main.dispatch task 7ccd1617-72c3-4165-9fdd-26eebcb5cc8d is finished
2021-05-30 11:01:36,487 DEBUG awx.main.dispatch task 00781c13-dd79-43e1-9796-5357ca6a30da starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:01:36,491 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:01:36,500 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:01:36,502 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:01:36,507 DEBUG awx.main.dispatch task 00781c13-dd79-43e1-9796-5357ca6a30da is finished
2021-05-30 11:01:56,331 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d206120d-692e-4440-bb2e-be291ce9aacb, queue=awx_private_queue)
[2021-05-30 11:01:56,331: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d206120d-692e-4440-bb2e-be291ce9aacb, queue=awx_private_queue)
2021-05-30 11:01:56,359 DEBUG awx.main.dispatch delivered d206120d-692e-4440-bb2e-be291ce9aacb to worker[181] qsize 0
2021-05-30 11:01:56,368 DEBUG awx.main.dispatch task d206120d-692e-4440-bb2e-be291ce9aacb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:01:56,371 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:01:56,386 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:01:56,389 DEBUG awx.main.dispatch task d206120d-692e-4440-bb2e-be291ce9aacb is finished
RESULT 2
OKREADY
2021-05-30 11:02:06,292 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(9e158ad3-a7e3-4b71-a548-feec6db216bd, queue=awx_private_queue)
[2021-05-30 11:02:06,292: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(9e158ad3-a7e3-4b71-a548-feec6db216bd, queue=awx_private_queue)
2021-05-30 11:02:06,309 DEBUG awx.main.dispatch delivered 9e158ad3-a7e3-4b71-a548-feec6db216bd to worker[181] qsize 0
2021-05-30 11:02:06,313 DEBUG awx.main.dispatch task 9e158ad3-a7e3-4b71-a548-feec6db216bd starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:02:06,323 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:02:06,327 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:02:06,329 DEBUG awx.main.dispatch task 9e158ad3-a7e3-4b71-a548-feec6db216bd is finished
2021-05-30 11:02:12,691 DEBUG awx.main.dispatch task eda1b0dc-9b80-4b5b-84cb-d44d11d68ab3 is finished
2021-05-30 11:02:16,325 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(96760e25-9cf9-442c-8635-d6baf8aa7857, queue=awx_private_queue)
[2021-05-30 11:02:16,325: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(96760e25-9cf9-442c-8635-d6baf8aa7857, queue=awx_private_queue)
2021-05-30 11:02:16,343 DEBUG awx.main.dispatch delivered 96760e25-9cf9-442c-8635-d6baf8aa7857 to worker[192] qsize 0
2021-05-30 11:02:16,344 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(96b1c74f-1b31-4b58-a847-bcce5d86e9af, queue=dev-awx-0)
[2021-05-30 11:02:16,344: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(96b1c74f-1b31-4b58-a847-bcce5d86e9af, queue=dev-awx-0)
2021-05-30 11:02:16,347 DEBUG awx.main.dispatch task 96760e25-9cf9-442c-8635-d6baf8aa7857 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:02:16,350 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:02:16,361 DEBUG awx.main.dispatch delivered 96b1c74f-1b31-4b58-a847-bcce5d86e9af to worker[182] qsize 0
2021-05-30 11:02:16,360 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:02:16,362 DEBUG awx.main.dispatch task 96760e25-9cf9-442c-8635-d6baf8aa7857 is finished
2021-05-30 11:02:16,363 DEBUG awx.main.dispatch task 96b1c74f-1b31-4b58-a847-bcce5d86e9af starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:02:16,383 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_1" does not exist
2021-05-30 11:02:16,388 DEBUG awx.main.dispatch task 96b1c74f-1b31-4b58-a847-bcce5d86e9af is finished
2021-05-30 11:02:36,292 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d5ace041-a66d-4317-af92-d85a79f6e356, queue=awx_private_queue)
[2021-05-30 11:02:36,292: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d5ace041-a66d-4317-af92-d85a79f6e356, queue=awx_private_queue)
2021-05-30 11:02:36,348 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3e638d30-90bc-4023-9d3d-aa9df37b7953, queue=dev-awx-0)
[2021-05-30 11:02:36,348: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3e638d30-90bc-4023-9d3d-aa9df37b7953, queue=dev-awx-0)
2021-05-30 11:02:36,347 DEBUG awx.main.dispatch delivered d5ace041-a66d-4317-af92-d85a79f6e356 to worker[181] qsize 0
2021-05-30 11:02:36,379 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2764f98a-e7b8-42fa-9b9c-c806b428e7a2, queue=awx_private_queue)
[2021-05-30 11:02:36,379: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2764f98a-e7b8-42fa-9b9c-c806b428e7a2, queue=awx_private_queue)
2021-05-30 11:02:36,391 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:02:36,387 DEBUG awx.main.dispatch task d5ace041-a66d-4317-af92-d85a79f6e356 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:02:36,403 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:02:36,412 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:02:36,417 DEBUG awx.main.dispatch task d5ace041-a66d-4317-af92-d85a79f6e356 is finished
2021-05-30 11:02:36,420 DEBUG awx.main.dispatch delivered 2764f98a-e7b8-42fa-9b9c-c806b428e7a2 to worker[183] qsize 0
2021-05-30 11:02:36,428 DEBUG awx.main.dispatch task 2764f98a-e7b8-42fa-9b9c-c806b428e7a2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:02:36,450 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:02:36,452 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:02:36,466 DEBUG awx.main.dispatch task 2764f98a-e7b8-42fa-9b9c-c806b428e7a2 is finished
2021-05-30 11:02:56,396 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(629e247c-54c6-45ec-80a5-ae1fa6c80293, queue=awx_private_queue)
[2021-05-30 11:02:56,396: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(629e247c-54c6-45ec-80a5-ae1fa6c80293, queue=awx_private_queue)
2021-05-30 11:02:56,443 DEBUG awx.main.dispatch delivered 629e247c-54c6-45ec-80a5-ae1fa6c80293 to worker[181] qsize 0
2021-05-30 11:02:56,455 DEBUG awx.main.dispatch task 629e247c-54c6-45ec-80a5-ae1fa6c80293 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:02:56,479 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:02:56,482 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:02:56,487 DEBUG awx.main.dispatch task 629e247c-54c6-45ec-80a5-ae1fa6c80293 is finished
RESULT 2
OKREADY
2021-05-30 11:03:06,293 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1dd83127-268f-43fc-b288-eb7ac37ff51a, queue=awx_private_queue)
[2021-05-30 11:03:06,293: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1dd83127-268f-43fc-b288-eb7ac37ff51a, queue=awx_private_queue)
2021-05-30 11:03:06,319 DEBUG awx.main.dispatch delivered 1dd83127-268f-43fc-b288-eb7ac37ff51a to worker[182] qsize 0
2021-05-30 11:03:06,332 DEBUG awx.main.dispatch task 1dd83127-268f-43fc-b288-eb7ac37ff51a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:03:06,364 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:03:06,368 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:03:06,375 DEBUG awx.main.dispatch task 1dd83127-268f-43fc-b288-eb7ac37ff51a is finished
2021-05-30 11:03:16,347 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(9d6a63b5-369e-48e7-bec4-3d8535c73a78, queue=dev-awx-0)
[2021-05-30 11:03:16,347: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(9d6a63b5-369e-48e7-bec4-3d8535c73a78, queue=dev-awx-0)
2021-05-30 11:03:16,386 DEBUG awx.main.dispatch delivered 9d6a63b5-369e-48e7-bec4-3d8535c73a78 to worker[182] qsize 0
2021-05-30 11:03:16,391 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b431aaed-9284-481c-ba9c-4efee387d418, queue=awx_private_queue)
[2021-05-30 11:03:16,391: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b431aaed-9284-481c-ba9c-4efee387d418, queue=awx_private_queue)
2021-05-30 11:03:16,398 DEBUG awx.main.dispatch task 9d6a63b5-369e-48e7-bec4-3d8535c73a78 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:03:16,421 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_2" does not exist
2021-05-30 11:03:16,425 DEBUG awx.main.dispatch task 9d6a63b5-369e-48e7-bec4-3d8535c73a78 is finished
2021-05-30 11:03:16,431 DEBUG awx.main.dispatch delivered b431aaed-9284-481c-ba9c-4efee387d418 to worker[181] qsize 0
2021-05-30 11:03:16,438 DEBUG awx.main.dispatch task b431aaed-9284-481c-ba9c-4efee387d418 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:03:16,450 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:03:16,451 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:03:16,455 DEBUG awx.main.dispatch task b431aaed-9284-481c-ba9c-4efee387d418 is finished
2021-05-30 11:03:36,307 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(66968d53-2914-47c2-80fe-5fdd017a470b, queue=awx_private_queue)
[2021-05-30 11:03:36,307: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(66968d53-2914-47c2-80fe-5fdd017a470b, queue=awx_private_queue)
2021-05-30 11:03:36,362 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(7b71342f-7423-4fbd-9ac9-9e331bd52e2a, queue=dev-awx-0)
[2021-05-30 11:03:36,362: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(7b71342f-7423-4fbd-9ac9-9e331bd52e2a, queue=dev-awx-0)
2021-05-30 11:03:36,361 DEBUG awx.main.dispatch delivered 66968d53-2914-47c2-80fe-5fdd017a470b to worker[183] qsize 0
2021-05-30 11:03:36,383 DEBUG awx.main.dispatch task 66968d53-2914-47c2-80fe-5fdd017a470b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:03:36,407 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:03:36,404 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e1f7e2ca-5bb9-4781-b804-32402873e09e, queue=awx_private_queue)
[2021-05-30 11:03:36,404: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e1f7e2ca-5bb9-4781-b804-32402873e09e, queue=awx_private_queue)
2021-05-30 11:03:36,409 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:03:36,403 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:03:36,413 DEBUG awx.main.dispatch task 66968d53-2914-47c2-80fe-5fdd017a470b is finished
2021-05-30 11:03:36,424 DEBUG awx.main.dispatch delivered e1f7e2ca-5bb9-4781-b804-32402873e09e to worker[183] qsize 0
2021-05-30 11:03:36,434 DEBUG awx.main.dispatch task e1f7e2ca-5bb9-4781-b804-32402873e09e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:03:36,446 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:03:36,448 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:03:36,452 DEBUG awx.main.dispatch task e1f7e2ca-5bb9-4781-b804-32402873e09e is finished
2021-05-30 11:03:56,397 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0f72e11c-5f6e-487e-ba03-34dbbdf54975, queue=awx_private_queue)
[2021-05-30 11:03:56,397: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0f72e11c-5f6e-487e-ba03-34dbbdf54975, queue=awx_private_queue)
2021-05-30 11:03:56,422 DEBUG awx.main.dispatch delivered 0f72e11c-5f6e-487e-ba03-34dbbdf54975 to worker[182] qsize 0
2021-05-30 11:03:56,432 DEBUG awx.main.dispatch task 0f72e11c-5f6e-487e-ba03-34dbbdf54975 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:03:56,449 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:03:56,451 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:03:56,467 DEBUG awx.main.dispatch task 0f72e11c-5f6e-487e-ba03-34dbbdf54975 is finished
RESULT 2
OKREADY
2021-05-30 11:04:06,303 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6d955abd-9543-4e3a-a18f-f40c2c924589, queue=awx_private_queue)
[2021-05-30 11:04:06,303: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6d955abd-9543-4e3a-a18f-f40c2c924589, queue=awx_private_queue)
2021-05-30 11:04:06,326 DEBUG awx.main.dispatch delivered 6d955abd-9543-4e3a-a18f-f40c2c924589 to worker[183] qsize 0
2021-05-30 11:04:06,335 DEBUG awx.main.dispatch task 6d955abd-9543-4e3a-a18f-f40c2c924589 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:04:06,347 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:04:06,349 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:04:06,354 DEBUG awx.main.dispatch task 6d955abd-9543-4e3a-a18f-f40c2c924589 is finished
2021-05-30 11:04:16,353 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5e813780-d348-4527-9634-89925cf598ae, queue=dev-awx-0)
[2021-05-30 11:04:16,353: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5e813780-d348-4527-9634-89925cf598ae, queue=dev-awx-0)
2021-05-30 11:04:16,396 DEBUG awx.main.dispatch delivered 5e813780-d348-4527-9634-89925cf598ae to worker[183] qsize 0
2021-05-30 11:04:16,404 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(323973d7-26c7-40cd-9c6e-b5802d12836a, queue=awx_private_queue)
[2021-05-30 11:04:16,404: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(323973d7-26c7-40cd-9c6e-b5802d12836a, queue=awx_private_queue)
2021-05-30 11:04:16,407 DEBUG awx.main.dispatch task 5e813780-d348-4527-9634-89925cf598ae starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:04:16,423 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_2" does not exist
2021-05-30 11:04:16,425 DEBUG awx.main.dispatch delivered 323973d7-26c7-40cd-9c6e-b5802d12836a to worker[181] qsize 0
2021-05-30 11:04:16,427 DEBUG awx.main.dispatch task 5e813780-d348-4527-9634-89925cf598ae is finished
2021-05-30 11:04:16,435 DEBUG awx.main.dispatch task 323973d7-26c7-40cd-9c6e-b5802d12836a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:04:16,462 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:04:16,463 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:04:16,469 DEBUG awx.main.dispatch task 323973d7-26c7-40cd-9c6e-b5802d12836a is finished
2021-05-30 11:04:36,317 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e2a8f495-a7cc-49de-b643-50921b6f4bcb, queue=awx_private_queue)
[2021-05-30 11:04:36,317: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e2a8f495-a7cc-49de-b643-50921b6f4bcb, queue=awx_private_queue)
2021-05-30 11:04:36,363 DEBUG awx.main.dispatch delivered e2a8f495-a7cc-49de-b643-50921b6f4bcb to worker[182] qsize 0
2021-05-30 11:04:36,363 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8af8b27a-eda0-4997-9db2-a89efe84a04c, queue=dev-awx-0)
[2021-05-30 11:04:36,363: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8af8b27a-eda0-4997-9db2-a89efe84a04c, queue=dev-awx-0)
2021-05-30 11:04:36,381 DEBUG awx.main.dispatch task e2a8f495-a7cc-49de-b643-50921b6f4bcb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:04:36,405 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:04:36,405 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(504595f9-dc11-479b-b2e4-286301f95d5a, queue=awx_private_queue)
[2021-05-30 11:04:36,405: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(504595f9-dc11-479b-b2e4-286301f95d5a, queue=awx_private_queue)
2021-05-30 11:04:36,408 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:04:36,403 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:04:36,414 DEBUG awx.main.dispatch task e2a8f495-a7cc-49de-b643-50921b6f4bcb is finished
2021-05-30 11:04:36,427 DEBUG awx.main.dispatch delivered 504595f9-dc11-479b-b2e4-286301f95d5a to worker[181] qsize 0
2021-05-30 11:04:36,433 DEBUG awx.main.dispatch task 504595f9-dc11-479b-b2e4-286301f95d5a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:04:36,442 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:04:36,443 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:04:36,447 DEBUG awx.main.dispatch task 504595f9-dc11-479b-b2e4-286301f95d5a is finished
2021-05-30 11:04:51,593 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(16441139-9b0f-458f-b592-1342cb98d9cd, queue=dev-awx-0)
[2021-05-30 11:04:51,593: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(16441139-9b0f-458f-b592-1342cb98d9cd, queue=dev-awx-0)
2021-05-30 11:04:51,618 DEBUG awx.main.dispatch delivered 16441139-9b0f-458f-b592-1342cb98d9cd to worker[181] qsize 0
2021-05-30 11:04:51,625 DEBUG awx.main.dispatch task 16441139-9b0f-458f-b592-1342cb98d9cd starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-30 11:04:51,637 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-30 11:04:51,645 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-30 11:04:51,650 DEBUG awx.main.dispatch task 16441139-9b0f-458f-b592-1342cb98d9cd is finished
2021-05-30 11:04:56,406 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ffc11e59-997c-4741-8cf4-32b8488d479d, queue=awx_private_queue)
[2021-05-30 11:04:56,406: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ffc11e59-997c-4741-8cf4-32b8488d479d, queue=awx_private_queue)
2021-05-30 11:04:56,435 DEBUG awx.main.dispatch delivered ffc11e59-997c-4741-8cf4-32b8488d479d to worker[192] qsize 0
2021-05-30 11:04:56,444 DEBUG awx.main.dispatch task ffc11e59-997c-4741-8cf4-32b8488d479d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:04:56,752 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:04:56,753 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:04:56,758 DEBUG awx.main.dispatch task ffc11e59-997c-4741-8cf4-32b8488d479d is finished
RESULT 2
OKREADY
2021-05-30 11:05:06,315 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(db522987-0b1c-4234-b502-538c989c31e6, queue=awx_private_queue)
[2021-05-30 11:05:06,315: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(db522987-0b1c-4234-b502-538c989c31e6, queue=awx_private_queue)
2021-05-30 11:05:06,348 DEBUG awx.main.dispatch delivered db522987-0b1c-4234-b502-538c989c31e6 to worker[182] qsize 0
2021-05-30 11:05:06,358 DEBUG awx.main.dispatch task db522987-0b1c-4234-b502-538c989c31e6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:05:06,390 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:05:06,392 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:05:06,397 DEBUG awx.main.dispatch task db522987-0b1c-4234-b502-538c989c31e6 is finished
2021-05-30 11:05:16,347 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0110ad3f-007c-4994-b93e-8a8963c1b2fd, queue=dev-awx-0)
[2021-05-30 11:05:16,347: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0110ad3f-007c-4994-b93e-8a8963c1b2fd, queue=dev-awx-0)
2021-05-30 11:05:16,373 DEBUG awx.main.dispatch delivered 0110ad3f-007c-4994-b93e-8a8963c1b2fd to worker[183] qsize 0
2021-05-30 11:05:16,384 DEBUG awx.main.dispatch task 0110ad3f-007c-4994-b93e-8a8963c1b2fd starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:05:16,404 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_3" does not exist
2021-05-30 11:05:16,409 DEBUG awx.main.dispatch task 0110ad3f-007c-4994-b93e-8a8963c1b2fd is finished
2021-05-30 11:05:16,405 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(83f61fae-84fd-408d-8656-f73db070d6db, queue=awx_private_queue)
[2021-05-30 11:05:16,405: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(83f61fae-84fd-408d-8656-f73db070d6db, queue=awx_private_queue)
2021-05-30 11:05:16,435 DEBUG awx.main.dispatch delivered 83f61fae-84fd-408d-8656-f73db070d6db to worker[181] qsize 0
2021-05-30 11:05:16,441 DEBUG awx.main.dispatch task 83f61fae-84fd-408d-8656-f73db070d6db starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:05:16,455 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:05:16,457 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:05:16,464 DEBUG awx.main.dispatch task 83f61fae-84fd-408d-8656-f73db070d6db is finished
2021-05-30 11:05:36,322 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a18b7937-9540-4806-a054-a875c424f581, queue=awx_private_queue)
[2021-05-30 11:05:36,322: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a18b7937-9540-4806-a054-a875c424f581, queue=awx_private_queue)
2021-05-30 11:05:36,361 DEBUG awx.main.dispatch delivered a18b7937-9540-4806-a054-a875c424f581 to worker[181] qsize 0
2021-05-30 11:05:36,372 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(18915b8f-46b9-4fe3-a937-625d1c07670a, queue=dev-awx-0)
[2021-05-30 11:05:36,372: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(18915b8f-46b9-4fe3-a937-625d1c07670a, queue=dev-awx-0)
2021-05-30 11:05:36,373 DEBUG awx.main.dispatch task a18b7937-9540-4806-a054-a875c424f581 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:05:36,387 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:05:36,390 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:05:36,395 DEBUG awx.main.dispatch task a18b7937-9540-4806-a054-a875c424f581 is finished
2021-05-30 11:05:36,405 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8ac8a637-5142-439f-8868-169ecb3928bc, queue=awx_private_queue)
[2021-05-30 11:05:36,405: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8ac8a637-5142-439f-8868-169ecb3928bc, queue=awx_private_queue)
2021-05-30 11:05:36,401 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:05:36,430 DEBUG awx.main.dispatch delivered 8ac8a637-5142-439f-8868-169ecb3928bc to worker[183] qsize 0
2021-05-30 11:05:36,438 DEBUG awx.main.dispatch task 8ac8a637-5142-439f-8868-169ecb3928bc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:05:36,450 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:05:36,453 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:05:36,457 DEBUG awx.main.dispatch task 8ac8a637-5142-439f-8868-169ecb3928bc is finished
2021-05-30 11:05:56,411 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8d465992-0567-4910-af8d-56ec6b39e564, queue=awx_private_queue)
[2021-05-30 11:05:56,411: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8d465992-0567-4910-af8d-56ec6b39e564, queue=awx_private_queue)
2021-05-30 11:05:56,449 DEBUG awx.main.dispatch delivered 8d465992-0567-4910-af8d-56ec6b39e564 to worker[182] qsize 0
2021-05-30 11:05:56,459 DEBUG awx.main.dispatch task 8d465992-0567-4910-af8d-56ec6b39e564 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:05:56,486 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:05:56,488 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:05:56,492 DEBUG awx.main.dispatch task 8d465992-0567-4910-af8d-56ec6b39e564 is finished
RESULT 2
OKREADY
2021-05-30 11:06:06,323 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(460b40df-230f-4aa8-b15a-aa71738fea8c, queue=awx_private_queue)
[2021-05-30 11:06:06,323: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(460b40df-230f-4aa8-b15a-aa71738fea8c, queue=awx_private_queue)
2021-05-30 11:06:06,356 DEBUG awx.main.dispatch delivered 460b40df-230f-4aa8-b15a-aa71738fea8c to worker[181] qsize 0
2021-05-30 11:06:06,364 DEBUG awx.main.dispatch task 460b40df-230f-4aa8-b15a-aa71738fea8c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:06:06,390 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:06:06,393 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:06:06,398 DEBUG awx.main.dispatch task 460b40df-230f-4aa8-b15a-aa71738fea8c is finished
2021-05-30 11:06:16,349 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(58d2b21c-bf7e-48b9-8c2a-6d0a1b2ead61, queue=dev-awx-0)
[2021-05-30 11:06:16,349: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(58d2b21c-bf7e-48b9-8c2a-6d0a1b2ead61, queue=dev-awx-0)
2021-05-30 11:06:16,379 DEBUG awx.main.dispatch delivered 58d2b21c-bf7e-48b9-8c2a-6d0a1b2ead61 to worker[192] qsize 0
2021-05-30 11:06:16,389 DEBUG awx.main.dispatch task 58d2b21c-bf7e-48b9-8c2a-6d0a1b2ead61 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:06:16,405 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(420d1f18-0b93-4510-86a0-730fc8a4e1ca, queue=awx_private_queue)
[2021-05-30 11:06:16,405: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(420d1f18-0b93-4510-86a0-730fc8a4e1ca, queue=awx_private_queue)
2021-05-30 11:06:16,413 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_1" does not exist
2021-05-30 11:06:16,418 DEBUG awx.main.dispatch task 58d2b21c-bf7e-48b9-8c2a-6d0a1b2ead61 is finished
2021-05-30 11:06:16,430 DEBUG awx.main.dispatch delivered 420d1f18-0b93-4510-86a0-730fc8a4e1ca to worker[182] qsize 0
2021-05-30 11:06:16,438 DEBUG awx.main.dispatch task 420d1f18-0b93-4510-86a0-730fc8a4e1ca starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:06:16,447 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:06:16,448 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:06:16,452 DEBUG awx.main.dispatch task 420d1f18-0b93-4510-86a0-730fc8a4e1ca is finished
2021-05-30 11:06:36,332 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(af52bb20-8ff7-46d7-81b4-a29115a4cab4, queue=awx_private_queue)
[2021-05-30 11:06:36,332: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(af52bb20-8ff7-46d7-81b4-a29115a4cab4, queue=awx_private_queue)
2021-05-30 11:06:36,357 DEBUG awx.main.dispatch delivered af52bb20-8ff7-46d7-81b4-a29115a4cab4 to worker[182] qsize 0
2021-05-30 11:06:36,365 DEBUG awx.main.dispatch task af52bb20-8ff7-46d7-81b4-a29115a4cab4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:06:36,379 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:06:36,381 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:06:36,370 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1bf63242-bc8f-4b9e-80b8-321caa62f62f, queue=dev-awx-0)
[2021-05-30 11:06:36,370: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1bf63242-bc8f-4b9e-80b8-321caa62f62f, queue=dev-awx-0)
2021-05-30 11:06:36,385 DEBUG awx.main.dispatch task af52bb20-8ff7-46d7-81b4-a29115a4cab4 is finished
2021-05-30 11:06:36,405 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a28ccc58-e4bb-4741-87bd-48268c65bc55, queue=awx_private_queue)
[2021-05-30 11:06:36,405: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a28ccc58-e4bb-4741-87bd-48268c65bc55, queue=awx_private_queue)
2021-05-30 11:06:36,404 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:06:36,427 DEBUG awx.main.dispatch delivered a28ccc58-e4bb-4741-87bd-48268c65bc55 to worker[183] qsize 0
2021-05-30 11:06:36,436 DEBUG awx.main.dispatch task a28ccc58-e4bb-4741-87bd-48268c65bc55 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:06:36,454 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:06:36,456 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:06:36,460 DEBUG awx.main.dispatch task a28ccc58-e4bb-4741-87bd-48268c65bc55 is finished
2021-05-30 11:06:56,413 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(93955411-4068-4c71-81df-c94c771cf268, queue=awx_private_queue)
[2021-05-30 11:06:56,413: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(93955411-4068-4c71-81df-c94c771cf268, queue=awx_private_queue)
2021-05-30 11:06:56,439 DEBUG awx.main.dispatch delivered 93955411-4068-4c71-81df-c94c771cf268 to worker[183] qsize 0
2021-05-30 11:06:56,449 DEBUG awx.main.dispatch task 93955411-4068-4c71-81df-c94c771cf268 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:06:56,460 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:06:56,461 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:06:56,465 DEBUG awx.main.dispatch task 93955411-4068-4c71-81df-c94c771cf268 is finished
RESULT 2
OKREADY
2021-05-30 11:07:06,336 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e3446af0-9997-4f18-ba7b-83cc4476ab34, queue=awx_private_queue)
[2021-05-30 11:07:06,336: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e3446af0-9997-4f18-ba7b-83cc4476ab34, queue=awx_private_queue)
2021-05-30 11:07:06,372 DEBUG awx.main.dispatch delivered e3446af0-9997-4f18-ba7b-83cc4476ab34 to worker[192] qsize 0
2021-05-30 11:07:06,386 DEBUG awx.main.dispatch task e3446af0-9997-4f18-ba7b-83cc4476ab34 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:07:06,406 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:07:06,409 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:07:06,414 DEBUG awx.main.dispatch task e3446af0-9997-4f18-ba7b-83cc4476ab34 is finished
2021-05-30 11:07:16,349 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(871a3007-e4d7-475b-9c74-eab12779458f, queue=dev-awx-0)
[2021-05-30 11:07:16,349: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(871a3007-e4d7-475b-9c74-eab12779458f, queue=dev-awx-0)
2021-05-30 11:07:16,381 DEBUG awx.main.dispatch delivered 871a3007-e4d7-475b-9c74-eab12779458f to worker[192] qsize 0
2021-05-30 11:07:16,389 DEBUG awx.main.dispatch task 871a3007-e4d7-475b-9c74-eab12779458f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:07:16,403 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_2" does not exist
2021-05-30 11:07:16,409 DEBUG awx.main.dispatch task 871a3007-e4d7-475b-9c74-eab12779458f is finished
2021-05-30 11:07:16,412 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f9020b41-a52a-490a-a088-9f7fce259ada, queue=awx_private_queue)
[2021-05-30 11:07:16,412: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f9020b41-a52a-490a-a088-9f7fce259ada, queue=awx_private_queue)
2021-05-30 11:07:16,436 DEBUG awx.main.dispatch delivered f9020b41-a52a-490a-a088-9f7fce259ada to worker[183] qsize 0
2021-05-30 11:07:16,444 DEBUG awx.main.dispatch task f9020b41-a52a-490a-a088-9f7fce259ada starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:07:16,462 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:07:16,464 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:07:16,469 DEBUG awx.main.dispatch task f9020b41-a52a-490a-a088-9f7fce259ada is finished
2021-05-30 11:07:36,343 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e9804158-724f-47b0-bca0-991d8171660c, queue=awx_private_queue)
[2021-05-30 11:07:36,343: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e9804158-724f-47b0-bca0-991d8171660c, queue=awx_private_queue)
2021-05-30 11:07:36,378 DEBUG awx.main.dispatch delivered e9804158-724f-47b0-bca0-991d8171660c to worker[182] qsize 0
2021-05-30 11:07:36,389 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(b8cb7e56-b7a9-44bc-bdea-22ae01a15bfb, queue=dev-awx-0)
[2021-05-30 11:07:36,389: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(b8cb7e56-b7a9-44bc-bdea-22ae01a15bfb, queue=dev-awx-0)
2021-05-30 11:07:36,389 DEBUG awx.main.dispatch task e9804158-724f-47b0-bca0-991d8171660c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:07:36,410 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:07:36,413 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:07:36,417 DEBUG awx.main.dispatch task e9804158-724f-47b0-bca0-991d8171660c is finished
2021-05-30 11:07:36,431 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(64286401-9355-4122-a4c0-886328eeb9d0, queue=awx_private_queue)
[2021-05-30 11:07:36,431: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(64286401-9355-4122-a4c0-886328eeb9d0, queue=awx_private_queue)
2021-05-30 11:07:36,431 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:07:36,458 DEBUG awx.main.dispatch delivered 64286401-9355-4122-a4c0-886328eeb9d0 to worker[182] qsize 0
2021-05-30 11:07:36,467 DEBUG awx.main.dispatch task 64286401-9355-4122-a4c0-886328eeb9d0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:07:36,477 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:07:36,479 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:07:36,484 DEBUG awx.main.dispatch task 64286401-9355-4122-a4c0-886328eeb9d0 is finished
2021-05-30 11:07:56,432 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(de9d8088-cf0d-4be5-a9ce-9b86da467d64, queue=awx_private_queue)
[2021-05-30 11:07:56,432: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(de9d8088-cf0d-4be5-a9ce-9b86da467d64, queue=awx_private_queue)
2021-05-30 11:07:56,461 DEBUG awx.main.dispatch delivered de9d8088-cf0d-4be5-a9ce-9b86da467d64 to worker[192] qsize 0
2021-05-30 11:07:56,476 DEBUG awx.main.dispatch task de9d8088-cf0d-4be5-a9ce-9b86da467d64 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:07:56,490 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:07:56,492 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:07:56,499 DEBUG awx.main.dispatch task de9d8088-cf0d-4be5-a9ce-9b86da467d64 is finished
RESULT 2
OKREADY
2021-05-30 11:08:06,342 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(69bea424-a2e1-4bd7-b3fb-b3f856f70282, queue=awx_private_queue)
[2021-05-30 11:08:06,342: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(69bea424-a2e1-4bd7-b3fb-b3f856f70282, queue=awx_private_queue)
2021-05-30 11:08:06,367 DEBUG awx.main.dispatch delivered 69bea424-a2e1-4bd7-b3fb-b3f856f70282 to worker[192] qsize 0
2021-05-30 11:08:06,377 DEBUG awx.main.dispatch task 69bea424-a2e1-4bd7-b3fb-b3f856f70282 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:08:06,389 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:08:06,391 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:08:06,396 DEBUG awx.main.dispatch task 69bea424-a2e1-4bd7-b3fb-b3f856f70282 is finished
2021-05-30 11:08:16,348 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3034c706-cd24-42b6-9240-5e52e98f8a55, queue=dev-awx-0)
[2021-05-30 11:08:16,348: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3034c706-cd24-42b6-9240-5e52e98f8a55, queue=dev-awx-0)
2021-05-30 11:08:16,374 DEBUG awx.main.dispatch delivered 3034c706-cd24-42b6-9240-5e52e98f8a55 to worker[181] qsize 0
2021-05-30 11:08:16,386 DEBUG awx.main.dispatch task 3034c706-cd24-42b6-9240-5e52e98f8a55 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:08:16,407 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_7" does not exist
2021-05-30 11:08:16,411 DEBUG awx.main.dispatch task 3034c706-cd24-42b6-9240-5e52e98f8a55 is finished
2021-05-30 11:08:16,432 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d1f626c2-fbc1-49a4-80e9-477594eebb6b, queue=awx_private_queue)
[2021-05-30 11:08:16,432: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d1f626c2-fbc1-49a4-80e9-477594eebb6b, queue=awx_private_queue)
2021-05-30 11:08:16,457 DEBUG awx.main.dispatch delivered d1f626c2-fbc1-49a4-80e9-477594eebb6b to worker[182] qsize 0
2021-05-30 11:08:16,471 DEBUG awx.main.dispatch task d1f626c2-fbc1-49a4-80e9-477594eebb6b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:08:16,485 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:08:16,487 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:08:16,491 DEBUG awx.main.dispatch task d1f626c2-fbc1-49a4-80e9-477594eebb6b is finished
2021-05-30 11:08:36,359 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e89c42c7-de21-4657-a46e-4eeecb63b241, queue=awx_private_queue)
[2021-05-30 11:08:36,359: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e89c42c7-de21-4657-a46e-4eeecb63b241, queue=awx_private_queue)
2021-05-30 11:08:36,407 DEBUG awx.main.dispatch delivered e89c42c7-de21-4657-a46e-4eeecb63b241 to worker[181] qsize 0
2021-05-30 11:08:36,416 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(58190317-887d-4921-ac39-83bbd5b1a3a2, queue=dev-awx-0)
[2021-05-30 11:08:36,416: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(58190317-887d-4921-ac39-83bbd5b1a3a2, queue=dev-awx-0)
2021-05-30 11:08:36,420 DEBUG awx.main.dispatch task e89c42c7-de21-4657-a46e-4eeecb63b241 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:08:36,438 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:08:36,441 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:08:36,447 DEBUG awx.main.dispatch task e89c42c7-de21-4657-a46e-4eeecb63b241 is finished
2021-05-30 11:08:36,459 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:08:36,460 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(76b63196-28af-4f02-bf08-adb5c49e38d9, queue=awx_private_queue)
[2021-05-30 11:08:36,460: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(76b63196-28af-4f02-bf08-adb5c49e38d9, queue=awx_private_queue)
2021-05-30 11:08:36,512 DEBUG awx.main.dispatch delivered 76b63196-28af-4f02-bf08-adb5c49e38d9 to worker[181] qsize 0
2021-05-30 11:08:36,519 DEBUG awx.main.dispatch task 76b63196-28af-4f02-bf08-adb5c49e38d9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:08:36,530 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:08:36,532 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:08:36,537 DEBUG awx.main.dispatch task 76b63196-28af-4f02-bf08-adb5c49e38d9 is finished
2021-05-30 11:08:56,468 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(207dd5bc-8a19-4077-8cdc-c9025a0113fe, queue=awx_private_queue)
[2021-05-30 11:08:56,468: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(207dd5bc-8a19-4077-8cdc-c9025a0113fe, queue=awx_private_queue)
2021-05-30 11:08:56,498 DEBUG awx.main.dispatch delivered 207dd5bc-8a19-4077-8cdc-c9025a0113fe to worker[183] qsize 0
2021-05-30 11:08:56,507 DEBUG awx.main.dispatch task 207dd5bc-8a19-4077-8cdc-c9025a0113fe starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:08:56,526 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:08:56,527 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:08:56,532 DEBUG awx.main.dispatch task 207dd5bc-8a19-4077-8cdc-c9025a0113fe is finished
RESULT 2
OKREADY
2021-05-30 11:09:06,352 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(14b8827d-6cf0-4853-b8cc-72314303b097, queue=awx_private_queue)
[2021-05-30 11:09:06,352: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(14b8827d-6cf0-4853-b8cc-72314303b097, queue=awx_private_queue)
2021-05-30 11:09:06,375 DEBUG awx.main.dispatch delivered 14b8827d-6cf0-4853-b8cc-72314303b097 to worker[182] qsize 0
2021-05-30 11:09:06,386 DEBUG awx.main.dispatch task 14b8827d-6cf0-4853-b8cc-72314303b097 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:09:06,415 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:09:06,418 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:09:06,424 DEBUG awx.main.dispatch task 14b8827d-6cf0-4853-b8cc-72314303b097 is finished
2021-05-30 11:09:16,350 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(5d16a9b2-91f0-4f23-94ca-fd7d0797b445, queue=dev-awx-0)
[2021-05-30 11:09:16,350: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(5d16a9b2-91f0-4f23-94ca-fd7d0797b445, queue=dev-awx-0)
2021-05-30 11:09:16,377 DEBUG awx.main.dispatch delivered 5d16a9b2-91f0-4f23-94ca-fd7d0797b445 to worker[181] qsize 0
2021-05-30 11:09:16,387 DEBUG awx.main.dispatch task 5d16a9b2-91f0-4f23-94ca-fd7d0797b445 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:09:16,403 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_8" does not exist
2021-05-30 11:09:16,408 DEBUG awx.main.dispatch task 5d16a9b2-91f0-4f23-94ca-fd7d0797b445 is finished
2021-05-30 11:09:16,465 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ccaba725-5cdc-44ee-9bab-19c86353d2ee, queue=awx_private_queue)
[2021-05-30 11:09:16,465: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ccaba725-5cdc-44ee-9bab-19c86353d2ee, queue=awx_private_queue)
2021-05-30 11:09:16,486 DEBUG awx.main.dispatch delivered ccaba725-5cdc-44ee-9bab-19c86353d2ee to worker[183] qsize 0
2021-05-30 11:09:16,492 DEBUG awx.main.dispatch task ccaba725-5cdc-44ee-9bab-19c86353d2ee starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:09:16,504 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:09:16,506 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:09:16,512 DEBUG awx.main.dispatch task ccaba725-5cdc-44ee-9bab-19c86353d2ee is finished
2021-05-30 11:09:36,362 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(082a2daf-6c2d-41a4-89ef-f1470163e1b0, queue=awx_private_queue)
[2021-05-30 11:09:36,362: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(082a2daf-6c2d-41a4-89ef-f1470163e1b0, queue=awx_private_queue)
2021-05-30 11:09:36,390 DEBUG awx.main.dispatch delivered 082a2daf-6c2d-41a4-89ef-f1470163e1b0 to worker[183] qsize 0
2021-05-30 11:09:36,397 DEBUG awx.main.dispatch task 082a2daf-6c2d-41a4-89ef-f1470163e1b0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:09:36,409 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:09:36,411 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:09:36,415 DEBUG awx.main.dispatch task 082a2daf-6c2d-41a4-89ef-f1470163e1b0 is finished
2021-05-30 11:09:36,416 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(155c5de8-fd6e-480b-b378-8f936af7ed09, queue=dev-awx-0)
[2021-05-30 11:09:36,416: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(155c5de8-fd6e-480b-b378-8f936af7ed09, queue=dev-awx-0)
2021-05-30 11:09:36,441 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:09:36,467 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(52abc309-b637-4d2d-beee-68edb0291f5a, queue=awx_private_queue)
[2021-05-30 11:09:36,467: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(52abc309-b637-4d2d-beee-68edb0291f5a, queue=awx_private_queue)
2021-05-30 11:09:36,490 DEBUG awx.main.dispatch delivered 52abc309-b637-4d2d-beee-68edb0291f5a to worker[183] qsize 0
2021-05-30 11:09:36,500 DEBUG awx.main.dispatch task 52abc309-b637-4d2d-beee-68edb0291f5a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:09:36,510 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:09:36,512 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:09:36,516 DEBUG awx.main.dispatch task 52abc309-b637-4d2d-beee-68edb0291f5a is finished
2021-05-30 11:09:56,474 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ec5e2bbc-2bc4-449b-b46c-f31dea071089, queue=awx_private_queue)
[2021-05-30 11:09:56,474: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ec5e2bbc-2bc4-449b-b46c-f31dea071089, queue=awx_private_queue)
2021-05-30 11:09:56,508 DEBUG awx.main.dispatch delivered ec5e2bbc-2bc4-449b-b46c-f31dea071089 to worker[192] qsize 0
2021-05-30 11:09:56,520 DEBUG awx.main.dispatch task ec5e2bbc-2bc4-449b-b46c-f31dea071089 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:09:56,543 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:09:56,545 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:09:56,549 DEBUG awx.main.dispatch task ec5e2bbc-2bc4-449b-b46c-f31dea071089 is finished
RESULT 2
OKREADY
2021-05-30 11:10:06,361 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d10c401a-ead0-4f26-baad-3601b4c29722, queue=awx_private_queue)
[2021-05-30 11:10:06,361: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d10c401a-ead0-4f26-baad-3601b4c29722, queue=awx_private_queue)
2021-05-30 11:10:06,406 DEBUG awx.main.dispatch delivered d10c401a-ead0-4f26-baad-3601b4c29722 to worker[183] qsize 0
2021-05-30 11:10:06,414 DEBUG awx.main.dispatch task d10c401a-ead0-4f26-baad-3601b4c29722 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:10:06,429 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:10:06,432 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:10:06,436 DEBUG awx.main.dispatch task d10c401a-ead0-4f26-baad-3601b4c29722 is finished
2021-05-30 11:10:16,352 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(803f4590-31a2-4d84-ab60-dfd20c0a17fd, queue=dev-awx-0)
[2021-05-30 11:10:16,352: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(803f4590-31a2-4d84-ab60-dfd20c0a17fd, queue=dev-awx-0)
2021-05-30 11:10:16,395 DEBUG awx.main.dispatch delivered 803f4590-31a2-4d84-ab60-dfd20c0a17fd to worker[192] qsize 0
2021-05-30 11:10:16,407 DEBUG awx.main.dispatch task 803f4590-31a2-4d84-ab60-dfd20c0a17fd starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:10:16,423 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_3" does not exist
2021-05-30 11:10:16,427 DEBUG awx.main.dispatch task 803f4590-31a2-4d84-ab60-dfd20c0a17fd is finished
2021-05-30 11:10:16,472 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d9089f74-87e8-45a8-af21-0be22908fd67, queue=awx_private_queue)
[2021-05-30 11:10:16,472: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d9089f74-87e8-45a8-af21-0be22908fd67, queue=awx_private_queue)
2021-05-30 11:10:16,509 DEBUG awx.main.dispatch delivered d9089f74-87e8-45a8-af21-0be22908fd67 to worker[181] qsize 0
2021-05-30 11:10:16,518 DEBUG awx.main.dispatch task d9089f74-87e8-45a8-af21-0be22908fd67 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:10:16,538 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:10:16,539 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:10:16,544 DEBUG awx.main.dispatch task d9089f74-87e8-45a8-af21-0be22908fd67 is finished
2021-05-30 11:10:36,371 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7f20a72a-1f00-4913-8cb2-a15025c32623, queue=awx_private_queue)
[2021-05-30 11:10:36,371: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7f20a72a-1f00-4913-8cb2-a15025c32623, queue=awx_private_queue)
2021-05-30 11:10:36,402 DEBUG awx.main.dispatch delivered 7f20a72a-1f00-4913-8cb2-a15025c32623 to worker[192] qsize 0
2021-05-30 11:10:36,410 DEBUG awx.main.dispatch task 7f20a72a-1f00-4913-8cb2-a15025c32623 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:10:36,422 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:10:36,416 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bdeed21f-8e14-47d6-bbab-3ee6f5ef0289, queue=dev-awx-0)
[2021-05-30 11:10:36,416: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bdeed21f-8e14-47d6-bbab-3ee6f5ef0289, queue=dev-awx-0)
2021-05-30 11:10:36,424 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:10:36,428 DEBUG awx.main.dispatch task 7f20a72a-1f00-4913-8cb2-a15025c32623 is finished
2021-05-30 11:10:36,447 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:10:36,472 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ab0cbf1c-7328-4e1b-b26b-326c6214d2ca, queue=awx_private_queue)
[2021-05-30 11:10:36,472: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ab0cbf1c-7328-4e1b-b26b-326c6214d2ca, queue=awx_private_queue)
2021-05-30 11:10:36,495 DEBUG awx.main.dispatch delivered ab0cbf1c-7328-4e1b-b26b-326c6214d2ca to worker[182] qsize 0
2021-05-30 11:10:36,518 DEBUG awx.main.dispatch task ab0cbf1c-7328-4e1b-b26b-326c6214d2ca starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:10:36,537 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:10:36,538 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:10:36,543 DEBUG awx.main.dispatch task ab0cbf1c-7328-4e1b-b26b-326c6214d2ca is finished
2021-05-30 11:10:56,485 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8ab4c17c-e912-43c8-99eb-8aa573046e45, queue=awx_private_queue)
[2021-05-30 11:10:56,485: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8ab4c17c-e912-43c8-99eb-8aa573046e45, queue=awx_private_queue)
2021-05-30 11:10:56,516 DEBUG awx.main.dispatch delivered 8ab4c17c-e912-43c8-99eb-8aa573046e45 to worker[183] qsize 0
2021-05-30 11:10:56,527 DEBUG awx.main.dispatch task 8ab4c17c-e912-43c8-99eb-8aa573046e45 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:10:56,545 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:10:56,547 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:10:56,555 DEBUG awx.main.dispatch task 8ab4c17c-e912-43c8-99eb-8aa573046e45 is finished
RESULT 2
OKREADY
2021-05-30 11:11:06,369 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a1b74368-304a-4d71-a8d5-ceb4954ec764, queue=awx_private_queue)
[2021-05-30 11:11:06,369: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a1b74368-304a-4d71-a8d5-ceb4954ec764, queue=awx_private_queue)
2021-05-30 11:11:06,399 DEBUG awx.main.dispatch delivered a1b74368-304a-4d71-a8d5-ceb4954ec764 to worker[181] qsize 0
2021-05-30 11:11:06,405 DEBUG awx.main.dispatch task a1b74368-304a-4d71-a8d5-ceb4954ec764 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:11:06,419 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:11:06,421 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:11:06,426 DEBUG awx.main.dispatch task a1b74368-304a-4d71-a8d5-ceb4954ec764 is finished
2021-05-30 11:11:16,349 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(30ad786e-089c-40dc-8673-00ce7ca87180, queue=dev-awx-0)
[2021-05-30 11:11:16,349: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(30ad786e-089c-40dc-8673-00ce7ca87180, queue=dev-awx-0)
2021-05-30 11:11:16,371 DEBUG awx.main.dispatch delivered 30ad786e-089c-40dc-8673-00ce7ca87180 to worker[181] qsize 0
2021-05-30 11:11:16,379 DEBUG awx.main.dispatch task 30ad786e-089c-40dc-8673-00ce7ca87180 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:11:16,390 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_9" does not exist
2021-05-30 11:11:16,394 DEBUG awx.main.dispatch task 30ad786e-089c-40dc-8673-00ce7ca87180 is finished
2021-05-30 11:11:16,484 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(6174e3b2-e2e4-4691-a88d-968b60735f93, queue=awx_private_queue)
[2021-05-30 11:11:16,484: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(6174e3b2-e2e4-4691-a88d-968b60735f93, queue=awx_private_queue)
2021-05-30 11:11:16,512 DEBUG awx.main.dispatch delivered 6174e3b2-e2e4-4691-a88d-968b60735f93 to worker[182] qsize 0
2021-05-30 11:11:16,519 DEBUG awx.main.dispatch task 6174e3b2-e2e4-4691-a88d-968b60735f93 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:11:16,529 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:11:16,531 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:11:16,536 DEBUG awx.main.dispatch task 6174e3b2-e2e4-4691-a88d-968b60735f93 is finished
2021-05-30 11:11:36,381 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f68ba1f2-439c-4a5c-9ee9-f51d22fb2b14, queue=awx_private_queue)
[2021-05-30 11:11:36,381: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f68ba1f2-439c-4a5c-9ee9-f51d22fb2b14, queue=awx_private_queue)
2021-05-30 11:11:36,410 DEBUG awx.main.dispatch delivered f68ba1f2-439c-4a5c-9ee9-f51d22fb2b14 to worker[181] qsize 0
2021-05-30 11:11:36,418 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e99f8fec-7058-4057-a1a9-21e0056567e0, queue=dev-awx-0)
[2021-05-30 11:11:36,418: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e99f8fec-7058-4057-a1a9-21e0056567e0, queue=dev-awx-0)
2021-05-30 11:11:36,419 DEBUG awx.main.dispatch task f68ba1f2-439c-4a5c-9ee9-f51d22fb2b14 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:11:36,437 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:11:36,442 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:11:36,448 DEBUG awx.main.dispatch task f68ba1f2-439c-4a5c-9ee9-f51d22fb2b14 is finished
2021-05-30 11:11:36,445 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:11:36,484 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(07b1730c-eef8-4742-98ce-3f5f9a33c52a, queue=awx_private_queue)
[2021-05-30 11:11:36,484: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(07b1730c-eef8-4742-98ce-3f5f9a33c52a, queue=awx_private_queue)
2021-05-30 11:11:36,510 DEBUG awx.main.dispatch delivered 07b1730c-eef8-4742-98ce-3f5f9a33c52a to worker[183] qsize 0
2021-05-30 11:11:36,533 DEBUG awx.main.dispatch task 07b1730c-eef8-4742-98ce-3f5f9a33c52a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:11:36,546 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:11:36,552 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:11:36,558 DEBUG awx.main.dispatch task 07b1730c-eef8-4742-98ce-3f5f9a33c52a is finished
2021-05-30 11:11:56,499 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(801c8601-eee8-40a8-aaac-e39dc2414216, queue=awx_private_queue)
[2021-05-30 11:11:56,499: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(801c8601-eee8-40a8-aaac-e39dc2414216, queue=awx_private_queue)
2021-05-30 11:11:56,544 DEBUG awx.main.dispatch delivered 801c8601-eee8-40a8-aaac-e39dc2414216 to worker[181] qsize 0
2021-05-30 11:11:56,557 DEBUG awx.main.dispatch task 801c8601-eee8-40a8-aaac-e39dc2414216 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:11:56,576 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:11:56,578 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:11:56,585 DEBUG awx.main.dispatch task 801c8601-eee8-40a8-aaac-e39dc2414216 is finished
RESULT 2
OKREADY
2021-05-30 11:12:06,380 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(1758653e-bde3-43c3-9832-c1ac3fe46c6d, queue=awx_private_queue)
[2021-05-30 11:12:06,380: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(1758653e-bde3-43c3-9832-c1ac3fe46c6d, queue=awx_private_queue)
2021-05-30 11:12:06,409 DEBUG awx.main.dispatch delivered 1758653e-bde3-43c3-9832-c1ac3fe46c6d to worker[192] qsize 0
2021-05-30 11:12:06,420 DEBUG awx.main.dispatch task 1758653e-bde3-43c3-9832-c1ac3fe46c6d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:12:06,451 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:12:06,455 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:12:06,463 DEBUG awx.main.dispatch task 1758653e-bde3-43c3-9832-c1ac3fe46c6d is finished
2021-05-30 11:12:16,348 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2abe6375-fafb-4b48-bc56-775b7fa368a2, queue=dev-awx-0)
[2021-05-30 11:12:16,348: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2abe6375-fafb-4b48-bc56-775b7fa368a2, queue=dev-awx-0)
2021-05-30 11:12:16,373 DEBUG awx.main.dispatch delivered 2abe6375-fafb-4b48-bc56-775b7fa368a2 to worker[183] qsize 0
2021-05-30 11:12:16,382 DEBUG awx.main.dispatch task 2abe6375-fafb-4b48-bc56-775b7fa368a2 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:12:16,403 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_4" does not exist
2021-05-30 11:12:16,407 DEBUG awx.main.dispatch task 2abe6375-fafb-4b48-bc56-775b7fa368a2 is finished
2021-05-30 11:12:16,494 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(099c6541-cc91-4b0c-9338-32c0c576f34b, queue=awx_private_queue)
[2021-05-30 11:12:16,494: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(099c6541-cc91-4b0c-9338-32c0c576f34b, queue=awx_private_queue)
2021-05-30 11:12:16,516 DEBUG awx.main.dispatch delivered 099c6541-cc91-4b0c-9338-32c0c576f34b to worker[182] qsize 0
2021-05-30 11:12:16,526 DEBUG awx.main.dispatch task 099c6541-cc91-4b0c-9338-32c0c576f34b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:12:16,542 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:12:16,544 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:12:16,548 DEBUG awx.main.dispatch task 099c6541-cc91-4b0c-9338-32c0c576f34b is finished
2021-05-30 11:12:36,389 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(97c00daf-afbc-4aae-982e-e1edbb9836db, queue=awx_private_queue)
[2021-05-30 11:12:36,389: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(97c00daf-afbc-4aae-982e-e1edbb9836db, queue=awx_private_queue)
2021-05-30 11:12:36,418 DEBUG awx.main.dispatch delivered 97c00daf-afbc-4aae-982e-e1edbb9836db to worker[192] qsize 0
2021-05-30 11:12:36,425 DEBUG awx.main.dispatch task 97c00daf-afbc-4aae-982e-e1edbb9836db starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:12:36,441 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:12:36,444 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:12:36,433 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3c48c8e5-df71-4698-b27c-155829426837, queue=dev-awx-0)
[2021-05-30 11:12:36,433: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3c48c8e5-df71-4698-b27c-155829426837, queue=dev-awx-0)
2021-05-30 11:12:36,451 DEBUG awx.main.dispatch task 97c00daf-afbc-4aae-982e-e1edbb9836db is finished
2021-05-30 11:12:36,481 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:12:36,494 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e4ed3d01-7384-4a0f-bd1c-92fbf1787e85, queue=awx_private_queue)
[2021-05-30 11:12:36,494: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e4ed3d01-7384-4a0f-bd1c-92fbf1787e85, queue=awx_private_queue)
2021-05-30 11:12:36,517 DEBUG awx.main.dispatch delivered e4ed3d01-7384-4a0f-bd1c-92fbf1787e85 to worker[192] qsize 0
2021-05-30 11:12:36,528 DEBUG awx.main.dispatch task e4ed3d01-7384-4a0f-bd1c-92fbf1787e85 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:12:36,545 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:12:36,547 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:12:36,551 DEBUG awx.main.dispatch task e4ed3d01-7384-4a0f-bd1c-92fbf1787e85 is finished
2021-05-30 11:12:56,506 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9f547a88-f6b3-4d9c-9bcd-e3c9563ab285, queue=awx_private_queue)
[2021-05-30 11:12:56,506: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9f547a88-f6b3-4d9c-9bcd-e3c9563ab285, queue=awx_private_queue)
2021-05-30 11:12:56,538 DEBUG awx.main.dispatch delivered 9f547a88-f6b3-4d9c-9bcd-e3c9563ab285 to worker[182] qsize 0
2021-05-30 11:12:56,545 DEBUG awx.main.dispatch task 9f547a88-f6b3-4d9c-9bcd-e3c9563ab285 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:12:56,559 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:12:56,561 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:12:56,568 DEBUG awx.main.dispatch task 9f547a88-f6b3-4d9c-9bcd-e3c9563ab285 is finished
RESULT 2
OKREADY
2021-05-30 11:13:06,389 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(b65412c6-acda-44e9-bf39-73ac03eb2986, queue=awx_private_queue)
[2021-05-30 11:13:06,389: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(b65412c6-acda-44e9-bf39-73ac03eb2986, queue=awx_private_queue)
2021-05-30 11:13:06,414 DEBUG awx.main.dispatch delivered b65412c6-acda-44e9-bf39-73ac03eb2986 to worker[181] qsize 0
2021-05-30 11:13:06,422 DEBUG awx.main.dispatch task b65412c6-acda-44e9-bf39-73ac03eb2986 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:13:06,439 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:13:06,441 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:13:06,446 DEBUG awx.main.dispatch task b65412c6-acda-44e9-bf39-73ac03eb2986 is finished
2021-05-30 11:13:16,349 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(886a6ea7-9866-4447-acfc-6c7eac85154a, queue=dev-awx-0)
[2021-05-30 11:13:16,349: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(886a6ea7-9866-4447-acfc-6c7eac85154a, queue=dev-awx-0)
2021-05-30 11:13:16,383 DEBUG awx.main.dispatch delivered 886a6ea7-9866-4447-acfc-6c7eac85154a to worker[181] qsize 0
2021-05-30 11:13:16,392 DEBUG awx.main.dispatch task 886a6ea7-9866-4447-acfc-6c7eac85154a starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:13:16,409 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_10" does not exist
2021-05-30 11:13:16,416 DEBUG awx.main.dispatch task 886a6ea7-9866-4447-acfc-6c7eac85154a is finished
2021-05-30 11:13:16,505 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c257cf33-186d-4807-a989-45141389319e, queue=awx_private_queue)
[2021-05-30 11:13:16,505: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c257cf33-186d-4807-a989-45141389319e, queue=awx_private_queue)
2021-05-30 11:13:16,531 DEBUG awx.main.dispatch delivered c257cf33-186d-4807-a989-45141389319e to worker[182] qsize 0
2021-05-30 11:13:16,537 DEBUG awx.main.dispatch task c257cf33-186d-4807-a989-45141389319e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:13:16,551 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:13:16,552 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:13:16,556 DEBUG awx.main.dispatch task c257cf33-186d-4807-a989-45141389319e is finished
2021-05-30 11:13:36,393 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a16c37c5-0947-4729-8586-958862e2755a, queue=awx_private_queue)
[2021-05-30 11:13:36,393: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a16c37c5-0947-4729-8586-958862e2755a, queue=awx_private_queue)
2021-05-30 11:13:36,430 DEBUG awx.main.dispatch delivered a16c37c5-0947-4729-8586-958862e2755a to worker[183] qsize 0
2021-05-30 11:13:36,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(976932b1-6982-45db-931f-c19b3941b4c2, queue=dev-awx-0)
[2021-05-30 11:13:36,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(976932b1-6982-45db-931f-c19b3941b4c2, queue=dev-awx-0)
2021-05-30 11:13:36,443 DEBUG awx.main.dispatch task a16c37c5-0947-4729-8586-958862e2755a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:13:36,462 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:13:36,465 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:13:36,471 DEBUG awx.main.dispatch task a16c37c5-0947-4729-8586-958862e2755a is finished
2021-05-30 11:13:36,468 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:13:36,504 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(271b7416-891d-43ad-a7d7-74faa9031f7d, queue=awx_private_queue)
[2021-05-30 11:13:36,504: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(271b7416-891d-43ad-a7d7-74faa9031f7d, queue=awx_private_queue)
2021-05-30 11:13:36,527 DEBUG awx.main.dispatch delivered 271b7416-891d-43ad-a7d7-74faa9031f7d to worker[182] qsize 0
2021-05-30 11:13:36,536 DEBUG awx.main.dispatch task 271b7416-891d-43ad-a7d7-74faa9031f7d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:13:36,549 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:13:36,550 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:13:36,559 DEBUG awx.main.dispatch task 271b7416-891d-43ad-a7d7-74faa9031f7d is finished
2021-05-30 11:13:56,520 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(99ba624c-35e8-4c44-a41c-0b9d91eadc60, queue=awx_private_queue)
[2021-05-30 11:13:56,520: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(99ba624c-35e8-4c44-a41c-0b9d91eadc60, queue=awx_private_queue)
2021-05-30 11:13:56,545 DEBUG awx.main.dispatch delivered 99ba624c-35e8-4c44-a41c-0b9d91eadc60 to worker[183] qsize 0
2021-05-30 11:13:56,557 DEBUG awx.main.dispatch task 99ba624c-35e8-4c44-a41c-0b9d91eadc60 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:13:56,584 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:13:56,586 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:13:56,591 DEBUG awx.main.dispatch task 99ba624c-35e8-4c44-a41c-0b9d91eadc60 is finished
RESULT 2
OKREADY
2021-05-30 11:14:06,391 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a4119355-5954-4d13-a7da-a2921e433b7d, queue=awx_private_queue)
[2021-05-30 11:14:06,391: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a4119355-5954-4d13-a7da-a2921e433b7d, queue=awx_private_queue)
2021-05-30 11:14:06,422 DEBUG awx.main.dispatch delivered a4119355-5954-4d13-a7da-a2921e433b7d to worker[183] qsize 0
2021-05-30 11:14:06,428 DEBUG awx.main.dispatch task a4119355-5954-4d13-a7da-a2921e433b7d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:14:06,438 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:14:06,440 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:14:06,443 DEBUG awx.main.dispatch task a4119355-5954-4d13-a7da-a2921e433b7d is finished
2021-05-30 11:14:16,349 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(032e4f52-5792-4ea1-995c-69e689175527, queue=dev-awx-0)
[2021-05-30 11:14:16,349: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(032e4f52-5792-4ea1-995c-69e689175527, queue=dev-awx-0)
2021-05-30 11:14:16,372 DEBUG awx.main.dispatch delivered 032e4f52-5792-4ea1-995c-69e689175527 to worker[192] qsize 0
2021-05-30 11:14:16,381 DEBUG awx.main.dispatch task 032e4f52-5792-4ea1-995c-69e689175527 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:14:16,407 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_4" does not exist
2021-05-30 11:14:16,412 DEBUG awx.main.dispatch task 032e4f52-5792-4ea1-995c-69e689175527 is finished
2021-05-30 11:14:16,519 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c15aa740-024a-44c1-92eb-b9a3484f6cd5, queue=awx_private_queue)
[2021-05-30 11:14:16,519: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c15aa740-024a-44c1-92eb-b9a3484f6cd5, queue=awx_private_queue)
2021-05-30 11:14:16,547 DEBUG awx.main.dispatch delivered c15aa740-024a-44c1-92eb-b9a3484f6cd5 to worker[181] qsize 0
2021-05-30 11:14:16,554 DEBUG awx.main.dispatch task c15aa740-024a-44c1-92eb-b9a3484f6cd5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:14:16,585 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:14:16,587 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:14:16,593 DEBUG awx.main.dispatch task c15aa740-024a-44c1-92eb-b9a3484f6cd5 is finished
2021-05-30 11:14:36,389 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7c056658-aa23-4cbe-9893-437dfd653af7, queue=awx_private_queue)
[2021-05-30 11:14:36,389: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7c056658-aa23-4cbe-9893-437dfd653af7, queue=awx_private_queue)
2021-05-30 11:14:36,410 DEBUG awx.main.dispatch delivered 7c056658-aa23-4cbe-9893-437dfd653af7 to worker[182] qsize 0
2021-05-30 11:14:36,417 DEBUG awx.main.dispatch task 7c056658-aa23-4cbe-9893-437dfd653af7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:14:36,431 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:14:36,433 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:14:36,440 DEBUG awx.main.dispatch task 7c056658-aa23-4cbe-9893-437dfd653af7 is finished
2021-05-30 11:14:36,443 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(811db413-f5d5-4bf1-82c8-e9716230b31e, queue=dev-awx-0)
[2021-05-30 11:14:36,443: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(811db413-f5d5-4bf1-82c8-e9716230b31e, queue=dev-awx-0)
2021-05-30 11:14:36,471 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:14:36,518 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(20cce99b-28f1-4a57-86fd-f9691b16545c, queue=awx_private_queue)
[2021-05-30 11:14:36,518: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(20cce99b-28f1-4a57-86fd-f9691b16545c, queue=awx_private_queue)
2021-05-30 11:14:36,538 DEBUG awx.main.dispatch delivered 20cce99b-28f1-4a57-86fd-f9691b16545c to worker[183] qsize 0
2021-05-30 11:14:36,545 DEBUG awx.main.dispatch task 20cce99b-28f1-4a57-86fd-f9691b16545c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:14:36,558 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:14:36,560 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:14:36,573 DEBUG awx.main.dispatch task 20cce99b-28f1-4a57-86fd-f9691b16545c is finished
2021-05-30 11:14:51,599 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(bb167cfe-1ae7-46b7-b9cc-5193c7c01368, queue=dev-awx-0)
[2021-05-30 11:14:51,599: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(bb167cfe-1ae7-46b7-b9cc-5193c7c01368, queue=dev-awx-0)
2021-05-30 11:14:51,634 DEBUG awx.main.dispatch delivered bb167cfe-1ae7-46b7-b9cc-5193c7c01368 to worker[183] qsize 0
2021-05-30 11:14:51,642 DEBUG awx.main.dispatch task bb167cfe-1ae7-46b7-b9cc-5193c7c01368 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-05-30 11:14:51,653 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-05-30 11:14:51,660 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 470, in awx_isolated_heartbeat
for isolated_instance in isolated_instance_qs:
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: ...pu_capacity", "main_instance"."mem_capacity" FROM "main_inst...
^
2021-05-30 11:14:51,664 DEBUG awx.main.dispatch task bb167cfe-1ae7-46b7-b9cc-5193c7c01368 is finished
2021-05-30 11:14:56,519 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4fe60409-1714-41ee-8259-677811114fe1, queue=awx_private_queue)
[2021-05-30 11:14:56,519: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4fe60409-1714-41ee-8259-677811114fe1, queue=awx_private_queue)
2021-05-30 11:14:56,545 DEBUG awx.main.dispatch delivered 4fe60409-1714-41ee-8259-677811114fe1 to worker[192] qsize 0
2021-05-30 11:14:56,554 DEBUG awx.main.dispatch task 4fe60409-1714-41ee-8259-677811114fe1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:14:56,565 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:14:56,568 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:14:56,572 DEBUG awx.main.dispatch task 4fe60409-1714-41ee-8259-677811114fe1 is finished
RESULT 2
OKREADY
2021-05-30 11:15:06,393 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a95fb498-5052-4b0d-a1f2-45bc26fa3a23, queue=awx_private_queue)
[2021-05-30 11:15:06,393: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a95fb498-5052-4b0d-a1f2-45bc26fa3a23, queue=awx_private_queue)
2021-05-30 11:15:06,432 DEBUG awx.main.dispatch delivered a95fb498-5052-4b0d-a1f2-45bc26fa3a23 to worker[182] qsize 0
2021-05-30 11:15:06,445 DEBUG awx.main.dispatch task a95fb498-5052-4b0d-a1f2-45bc26fa3a23 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:15:06,466 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:15:06,470 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:15:06,474 DEBUG awx.main.dispatch task a95fb498-5052-4b0d-a1f2-45bc26fa3a23 is finished
2021-05-30 11:15:16,353 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(658cf0f0-3460-4f42-afb6-7a9720acf005, queue=dev-awx-0)
[2021-05-30 11:15:16,353: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(658cf0f0-3460-4f42-afb6-7a9720acf005, queue=dev-awx-0)
2021-05-30 11:15:16,402 DEBUG awx.main.dispatch delivered 658cf0f0-3460-4f42-afb6-7a9720acf005 to worker[183] qsize 0
2021-05-30 11:15:16,412 DEBUG awx.main.dispatch task 658cf0f0-3460-4f42-afb6-7a9720acf005 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:15:16,428 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_5" does not exist
2021-05-30 11:15:16,432 DEBUG awx.main.dispatch task 658cf0f0-3460-4f42-afb6-7a9720acf005 is finished
2021-05-30 11:15:16,519 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(9eaadc48-82f5-48e9-8acb-8239e8c5e517, queue=awx_private_queue)
[2021-05-30 11:15:16,519: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(9eaadc48-82f5-48e9-8acb-8239e8c5e517, queue=awx_private_queue)
2021-05-30 11:15:16,560 DEBUG awx.main.dispatch delivered 9eaadc48-82f5-48e9-8acb-8239e8c5e517 to worker[183] qsize 0
2021-05-30 11:15:16,568 DEBUG awx.main.dispatch task 9eaadc48-82f5-48e9-8acb-8239e8c5e517 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:15:16,579 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:15:16,580 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:15:16,586 DEBUG awx.main.dispatch task 9eaadc48-82f5-48e9-8acb-8239e8c5e517 is finished
2021-05-30 11:15:36,391 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(33130cc9-fab5-4925-b7a1-c00f704f5510, queue=awx_private_queue)
[2021-05-30 11:15:36,391: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(33130cc9-fab5-4925-b7a1-c00f704f5510, queue=awx_private_queue)
2021-05-30 11:15:36,418 DEBUG awx.main.dispatch delivered 33130cc9-fab5-4925-b7a1-c00f704f5510 to worker[181] qsize 0
2021-05-30 11:15:36,430 DEBUG awx.main.dispatch task 33130cc9-fab5-4925-b7a1-c00f704f5510 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:15:36,449 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:15:36,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(c4c054ec-1181-46a3-a847-8c066bc60a60, queue=dev-awx-0)
[2021-05-30 11:15:36,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(c4c054ec-1181-46a3-a847-8c066bc60a60, queue=dev-awx-0)
2021-05-30 11:15:36,452 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:15:36,456 DEBUG awx.main.dispatch task 33130cc9-fab5-4925-b7a1-c00f704f5510 is finished
2021-05-30 11:15:36,486 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:15:36,519 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cc330aab-c5c2-4df3-aa1d-0716238f8deb, queue=awx_private_queue)
[2021-05-30 11:15:36,519: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cc330aab-c5c2-4df3-aa1d-0716238f8deb, queue=awx_private_queue)
2021-05-30 11:15:36,553 DEBUG awx.main.dispatch delivered cc330aab-c5c2-4df3-aa1d-0716238f8deb to worker[181] qsize 0
2021-05-30 11:15:36,564 DEBUG awx.main.dispatch task cc330aab-c5c2-4df3-aa1d-0716238f8deb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:15:36,575 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:15:36,577 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:15:36,582 DEBUG awx.main.dispatch task cc330aab-c5c2-4df3-aa1d-0716238f8deb is finished
2021-05-30 11:15:56,530 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2071705d-817f-485e-98e3-d1f35d9d22d5, queue=awx_private_queue)
[2021-05-30 11:15:56,530: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2071705d-817f-485e-98e3-d1f35d9d22d5, queue=awx_private_queue)
2021-05-30 11:15:56,555 DEBUG awx.main.dispatch delivered 2071705d-817f-485e-98e3-d1f35d9d22d5 to worker[183] qsize 0
2021-05-30 11:15:56,562 DEBUG awx.main.dispatch task 2071705d-817f-485e-98e3-d1f35d9d22d5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:15:56,583 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:15:56,584 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:15:56,589 DEBUG awx.main.dispatch task 2071705d-817f-485e-98e3-d1f35d9d22d5 is finished
RESULT 2
OKREADY
2021-05-30 11:16:06,391 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(772e3e31-0530-485c-bd7b-0656fc0ef3ed, queue=awx_private_queue)
[2021-05-30 11:16:06,391: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(772e3e31-0530-485c-bd7b-0656fc0ef3ed, queue=awx_private_queue)
2021-05-30 11:16:06,415 DEBUG awx.main.dispatch delivered 772e3e31-0530-485c-bd7b-0656fc0ef3ed to worker[181] qsize 0
2021-05-30 11:16:06,422 DEBUG awx.main.dispatch task 772e3e31-0530-485c-bd7b-0656fc0ef3ed starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:16:06,435 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:16:06,437 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:16:06,441 DEBUG awx.main.dispatch task 772e3e31-0530-485c-bd7b-0656fc0ef3ed is finished
2021-05-30 11:16:16,350 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(56259567-4ace-4740-ae4a-37a6dcf30351, queue=dev-awx-0)
[2021-05-30 11:16:16,350: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(56259567-4ace-4740-ae4a-37a6dcf30351, queue=dev-awx-0)
2021-05-30 11:16:16,375 DEBUG awx.main.dispatch delivered 56259567-4ace-4740-ae4a-37a6dcf30351 to worker[183] qsize 0
2021-05-30 11:16:16,387 DEBUG awx.main.dispatch task 56259567-4ace-4740-ae4a-37a6dcf30351 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:16:16,400 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_6" does not exist
2021-05-30 11:16:16,404 DEBUG awx.main.dispatch task 56259567-4ace-4740-ae4a-37a6dcf30351 is finished
2021-05-30 11:16:16,529 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(988b25af-80f6-44ea-b97d-219693b21429, queue=awx_private_queue)
[2021-05-30 11:16:16,529: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(988b25af-80f6-44ea-b97d-219693b21429, queue=awx_private_queue)
2021-05-30 11:16:16,559 DEBUG awx.main.dispatch delivered 988b25af-80f6-44ea-b97d-219693b21429 to worker[183] qsize 0
2021-05-30 11:16:16,567 DEBUG awx.main.dispatch task 988b25af-80f6-44ea-b97d-219693b21429 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:16:16,578 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:16:16,580 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:16:16,584 DEBUG awx.main.dispatch task 988b25af-80f6-44ea-b97d-219693b21429 is finished
2021-05-30 11:16:36,405 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d0af37cb-4ccc-4d40-9960-41dd0af3b84e, queue=awx_private_queue)
[2021-05-30 11:16:36,405: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d0af37cb-4ccc-4d40-9960-41dd0af3b84e, queue=awx_private_queue)
2021-05-30 11:16:36,438 DEBUG awx.main.dispatch delivered d0af37cb-4ccc-4d40-9960-41dd0af3b84e to worker[183] qsize 0
2021-05-30 11:16:36,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(11b68c73-9a34-4487-9e9a-7ee126550b40, queue=dev-awx-0)
[2021-05-30 11:16:36,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(11b68c73-9a34-4487-9e9a-7ee126550b40, queue=dev-awx-0)
2021-05-30 11:16:36,450 DEBUG awx.main.dispatch task d0af37cb-4ccc-4d40-9960-41dd0af3b84e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:16:36,466 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:16:36,469 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:16:36,475 DEBUG awx.main.dispatch task d0af37cb-4ccc-4d40-9960-41dd0af3b84e is finished
2021-05-30 11:16:36,478 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
2021-05-30 11:16:36,529 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a38f93d7-09b8-4309-9ab0-cfe65baff1d2, queue=awx_private_queue)
[2021-05-30 11:16:36,529: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a38f93d7-09b8-4309-9ab0-cfe65baff1d2, queue=awx_private_queue)
2021-05-30 11:16:36,555 DEBUG awx.main.dispatch delivered a38f93d7-09b8-4309-9ab0-cfe65baff1d2 to worker[181] qsize 0
2021-05-30 11:16:36,565 DEBUG awx.main.dispatch task a38f93d7-09b8-4309-9ab0-cfe65baff1d2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:16:36,586 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:16:36,588 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:16:36,593 DEBUG awx.main.dispatch task a38f93d7-09b8-4309-9ab0-cfe65baff1d2 is finished
2021-05-30 11:16:56,541 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7a2d3808-a4d8-4220-ba57-51e14255ae34, queue=awx_private_queue)
[2021-05-30 11:16:56,541: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7a2d3808-a4d8-4220-ba57-51e14255ae34, queue=awx_private_queue)
2021-05-30 11:16:56,596 DEBUG awx.main.dispatch delivered 7a2d3808-a4d8-4220-ba57-51e14255ae34 to worker[182] qsize 0
2021-05-30 11:16:56,614 DEBUG awx.main.dispatch task 7a2d3808-a4d8-4220-ba57-51e14255ae34 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:16:56,633 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:16:56,635 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:16:56,639 DEBUG awx.main.dispatch task 7a2d3808-a4d8-4220-ba57-51e14255ae34 is finished
RESULT 2
OKREADY
2021-05-30 11:17:06,402 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6d98eb9a-7fe1-4a5d-a428-d2ed40f3bc8c, queue=awx_private_queue)
[2021-05-30 11:17:06,402: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6d98eb9a-7fe1-4a5d-a428-d2ed40f3bc8c, queue=awx_private_queue)
2021-05-30 11:17:06,426 DEBUG awx.main.dispatch delivered 6d98eb9a-7fe1-4a5d-a428-d2ed40f3bc8c to worker[181] qsize 0
2021-05-30 11:17:06,433 DEBUG awx.main.dispatch task 6d98eb9a-7fe1-4a5d-a428-d2ed40f3bc8c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:17:06,445 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:17:06,447 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:17:06,451 DEBUG awx.main.dispatch task 6d98eb9a-7fe1-4a5d-a428-d2ed40f3bc8c is finished
2021-05-30 11:17:16,350 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3030ddf4-0517-4987-b3fc-51354a3fc8bc, queue=dev-awx-0)
[2021-05-30 11:17:16,350: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3030ddf4-0517-4987-b3fc-51354a3fc8bc, queue=dev-awx-0)
2021-05-30 11:17:16,374 DEBUG awx.main.dispatch delivered 3030ddf4-0517-4987-b3fc-51354a3fc8bc to worker[192] qsize 0
2021-05-30 11:17:16,386 DEBUG awx.main.dispatch task 3030ddf4-0517-4987-b3fc-51354a3fc8bc starting awx.main.tasks.awx_k8s_reaper(*[])
2021-05-30 11:17:16,412 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1103, in execute_sql
cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139789344183872_5" does not exist
2021-05-30 11:17:16,418 DEBUG awx.main.dispatch task 3030ddf4-0517-4987-b3fc-51354a3fc8bc is finished
2021-05-30 11:17:16,539 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(aebd4285-01fd-48fb-8e5f-f782ac28ebe6, queue=awx_private_queue)
[2021-05-30 11:17:16,539: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(aebd4285-01fd-48fb-8e5f-f782ac28ebe6, queue=awx_private_queue)
2021-05-30 11:17:16,561 DEBUG awx.main.dispatch delivered aebd4285-01fd-48fb-8e5f-f782ac28ebe6 to worker[182] qsize 0
2021-05-30 11:17:16,568 DEBUG awx.main.dispatch task aebd4285-01fd-48fb-8e5f-f782ac28ebe6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-05-30 11:17:16,586 DEBUG awx.main.scheduler Running Tower task manager.
2021-05-30 11:17:16,588 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instancegroup" does not exist
LINE 1: ... "main_instancegroup"."policy_instance_list" FROM "main_inst...
^
2021-05-30 11:17:16,593 DEBUG awx.main.dispatch task aebd4285-01fd-48fb-8e5f-f782ac28ebe6 is finished
2021-05-30 11:17:36,409 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(92587103-04c3-4a49-be67-8aa39fb1d7c6, queue=awx_private_queue)
[2021-05-30 11:17:36,409: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(92587103-04c3-4a49-be67-8aa39fb1d7c6, queue=awx_private_queue)
2021-05-30 11:17:36,433 DEBUG awx.main.dispatch delivered 92587103-04c3-4a49-be67-8aa39fb1d7c6 to worker[192] qsize 0
2021-05-30 11:17:36,440 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(fd6a6521-55f7-43d6-82ef-e41fdaff55a3, queue=dev-awx-0)
[2021-05-30 11:17:36,440: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(fd6a6521-55f7-43d6-82ef-e41fdaff55a3, queue=dev-awx-0)
2021-05-30 11:17:36,439 DEBUG awx.main.dispatch task 92587103-04c3-4a49-be67-8aa39fb1d7c6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-05-30 11:17:36,454 DEBUG awx.main.tasks Starting periodic scheduler
2021-05-30 11:17:36,458 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 489, in awx_periodic_scheduler
state = TowerScheduleState.get_solo()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/solo/models.py", line 54, in get_solo
obj, created = cls.objects.get_or_create(pk=cls.singleton_instance_id)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_towerschedulestate" does not exist
LINE 1: ...main_towerschedulestate"."schedule_last_run" FROM "main_towe...
^
2021-05-30 11:17:36,464 DEBUG awx.main.dispatch task 92587103-04c3-4a49-be67-8aa39fb1d7c6 is finished
2021-05-30 11:17:36,479 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 125, in register
if instance.exists():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 766, in exists
return self.query.has_results(using=self.db)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
return compiler.has_results()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
return bool(self.execute_sql(SINGLE))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise d