By type
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"elapsed": 0,
"match_groupdict": {},
"match_groups": [],
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"msg": "argument port is of type and we were unable to convert to int: cannot be converted to an int"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"db": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
Default organization added.
Demo Credential, Inventory, and Job Template added.
(changed: True)
2021-09-27 11:43:08,785 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(d3c3a213-1c8d-46ab-8e3c-9c502279a7b0, queue=awx_private_queue)
Successfully registered instance dev-awx-0
(changed: True)
2021-09-27 11:43:15,066 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(db91a27c-982a-42cf-a65c-9b7df086aacb, queue=awx_private_queue)
2021-09-27 11:43:15,251 DEBUG awx.main.dispatch publish awx.main.tasks.apply_cluster_membership_policies(b0c1acdb-bd84-4c2c-beb7-f6ae94bb54b1, queue=awx_private_queue)
Creating instance group tower
(changed: True)
/usr/bin/launch_awx_task.sh: line 27: unset: `#!/bin/sh': not a valid identifier
2021-09-27 11:43:16,068 INFO RPC interface 'supervisor' initialized
2021-09-27 11:43:16,068 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-09-27 11:43:16,069 INFO supervisord started with pid 150
2021-09-27 11:43:17,073 INFO spawned: 'awx-config-watcher' with pid 153
2021-09-27 11:43:17,078 INFO spawned: 'dispatcher' with pid 154
2021-09-27 11:43:17,083 INFO spawned: 'callback-receiver' with pid 155
2021-09-27 11:43:17,087 INFO spawned: 'channels-worker' with pid 156
READY
2021-09-27 11:43:18,235 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-09-27 11:43:18,235 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-09-27 11:43:18,235 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-09-27 11:43:18,235 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-09-27 11:43:20,411 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-09-27 11:43:20,411 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:175
2021-09-27 11:43:20,416 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-09-27 11:43:20,416 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:176
2021-09-27 11:43:20,421 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-09-27 11:43:20,421 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:178
2021-09-27 11:43:20,427 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-09-27 11:43:20,427 WARNING awx.main.commands.run_callback_receiver scaling up worker pid:179
2021-09-27 11:43:20,467 ERROR celery.beat Removing corrupted schedule file '/var/lib/awx/beat.db': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 485, in setup_schedule
self._store = self._open_schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/celery/beat.py", line 475, in _open_schedule
return self.persistence.open(self.schedule_filename, writeback=True)
File "/usr/lib64/python3.6/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib64/python3.6/shelve.py", line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib64/python3.6/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
2021-09-27 11:43:20,493 WARNING awx.main.dispatch scaling up worker pid:180
2021-09-27 11:43:20,501 WARNING awx.main.dispatch scaling up worker pid:181
2021-09-27 11:43:20,509 WARNING awx.main.dispatch scaling up worker pid:182
2021-09-27 11:43:20,517 WARNING awx.main.dispatch scaling up worker pid:183
2021-09-27 11:43:20,521 DEBUG awx.main.tasks Syncing Schedules
2021-09-27 11:43:20,610 - INFO - runworker - Using single-threaded worker.
2021-09-27 11:43:20,610 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2021-09-27 11:43:20,612 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2021-09-27 11:43:20,898 DEBUG awx.main.tasks Waited 0.013499259948730469 seconds to obtain lock name: cluster_policy_lock
2021-09-27 11:43:20,925 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2021-09-27 11:43:20,925 DEBUG awx.main.tasks Policy percentage, adding Instances [1] to Group tower
2021-09-27 11:43:20,926 DEBUG awx.main.tasks Adding instances [1] to group tower
2021-09-27 11:43:20,937 DEBUG awx.main.tasks Cluster policy computation finished in 0.0384061336517334 seconds
2021-09-27 11:43:20,942 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:43:20,993 DEBUG awx.main.dispatch listening on ['awx_private_queue [direct]', 'dev-awx-0 [direct]', 'tower_broadcast_all_dev-awx-0 [fanout]']
2021-09-27 11:43:40,617 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(82dce742-b7ef-42aa-9cbb-95cfe5cc1785, queue=awx_private_queue)
[2021-09-27 11:43:40,617: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(82dce742-b7ef-42aa-9cbb-95cfe5cc1785, queue=awx_private_queue)
2021-09-27 11:43:40,641 DEBUG awx.main.dispatch delivered 82dce742-b7ef-42aa-9cbb-95cfe5cc1785 to worker[180] qsize 0
2021-09-27 11:43:40,647 DEBUG awx.main.dispatch task 82dce742-b7ef-42aa-9cbb-95cfe5cc1785 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:43:40,651 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:43:40,718 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:43:40,890 DEBUG awx.main.dispatch task 82dce742-b7ef-42aa-9cbb-95cfe5cc1785 is finished
2021-09-27 11:43:50,581 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d88d1ad7-09ef-46fb-bd29-ba51b35d576d, queue=awx_private_queue)
[2021-09-27 11:43:50,581: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d88d1ad7-09ef-46fb-bd29-ba51b35d576d, queue=awx_private_queue)
2021-09-27 11:43:50,601 DEBUG awx.main.dispatch delivered d88d1ad7-09ef-46fb-bd29-ba51b35d576d to worker[183] qsize 0
2021-09-27 11:43:50,605 DEBUG awx.main.dispatch task d88d1ad7-09ef-46fb-bd29-ba51b35d576d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:43:57,974 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:43:58,063 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:43:58.055812+00:00
2021-09-27 11:43:58,364 DEBUG awx.main.dispatch task d88d1ad7-09ef-46fb-bd29-ba51b35d576d is finished
2021-09-27 11:44:00,594 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c7a5c3f8-095c-4815-8521-95afea6999ac, queue=awx_private_queue)
[2021-09-27 11:44:00,594: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c7a5c3f8-095c-4815-8521-95afea6999ac, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 11:44:00,616 DEBUG awx.main.dispatch delivered c7a5c3f8-095c-4815-8521-95afea6999ac to worker[183] qsize 0
2021-09-27 11:44:00,620 DEBUG awx.main.dispatch task c7a5c3f8-095c-4815-8521-95afea6999ac starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:44:00,622 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:44:01,444 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:44:02,377 DEBUG awx.main.dispatch task c7a5c3f8-095c-4815-8521-95afea6999ac is finished
2021-09-27 11:44:20,598 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(7ce344b7-bbb2-4013-a587-d67b4e7b5c17, queue=dev-awx-0)
[2021-09-27 11:44:20,598: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(7ce344b7-bbb2-4013-a587-d67b4e7b5c17, queue=dev-awx-0)
2021-09-27 11:44:20,944 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(47416186-6c99-4e83-aa3f-97b442a93d45, queue=dev-awx-0)
[2021-09-27 11:44:20,944: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(47416186-6c99-4e83-aa3f-97b442a93d45, queue=dev-awx-0)
2021-09-27 11:44:20,960 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ad40413f-8ce5-4cc9-b1c8-d69e28274e94, queue=awx_private_queue)
[2021-09-27 11:44:20,960: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ad40413f-8ce5-4cc9-b1c8-d69e28274e94, queue=awx_private_queue)
2021-09-27 11:44:20,965 DEBUG awx.main.dispatch delivered 7ce344b7-bbb2-4013-a587-d67b4e7b5c17 to worker[180] qsize 0
2021-09-27 11:44:20,967 DEBUG awx.main.dispatch delivered 47416186-6c99-4e83-aa3f-97b442a93d45 to worker[181] qsize 0
2021-09-27 11:44:20,967 DEBUG awx.main.dispatch task 7ce344b7-bbb2-4013-a587-d67b4e7b5c17 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:44:20,970 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:44:20,970 DEBUG awx.main.dispatch task 47416186-6c99-4e83-aa3f-97b442a93d45 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:44:20,979 DEBUG awx.main.dispatch delivered ad40413f-8ce5-4cc9-b1c8-d69e28274e94 to worker[182] qsize 0
2021-09-27 11:44:20,980 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cdad91d6-2ec7-4c6a-9778-e7d11cd55963, queue=awx_private_queue)
[2021-09-27 11:44:20,980: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cdad91d6-2ec7-4c6a-9778-e7d11cd55963, queue=awx_private_queue)
2021-09-27 11:44:20,982 DEBUG awx.main.dispatch task ad40413f-8ce5-4cc9-b1c8-d69e28274e94 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:44:20,998 DEBUG awx.main.dispatch delivered cdad91d6-2ec7-4c6a-9778-e7d11cd55963 to worker[183] qsize 0
2021-09-27 11:44:21,000 DEBUG awx.main.dispatch task cdad91d6-2ec7-4c6a-9778-e7d11cd55963 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:44:21,003 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:44:21,031 DEBUG awx.main.dispatch task 47416186-6c99-4e83-aa3f-97b442a93d45 is finished
2021-09-27 11:44:21,076 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:44:21,099 DEBUG awx.main.dispatch task 7ce344b7-bbb2-4013-a587-d67b4e7b5c17 is finished
2021-09-27 11:44:21,170 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:43:57.974801+00:00
2021-09-27 11:44:21,211 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:44:21,251 DEBUG awx.main.dispatch task ad40413f-8ce5-4cc9-b1c8-d69e28274e94 is finished
2021-09-27 11:44:21,282 DEBUG awx.main.dispatch task cdad91d6-2ec7-4c6a-9778-e7d11cd55963 is finished
2021-09-27 11:44:40,992 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(198b9894-11da-4261-a65e-a3e22e777b05, queue=awx_private_queue)
[2021-09-27 11:44:40,992: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(198b9894-11da-4261-a65e-a3e22e777b05, queue=awx_private_queue)
2021-09-27 11:44:41,008 DEBUG awx.main.dispatch delivered 198b9894-11da-4261-a65e-a3e22e777b05 to worker[180] qsize 0
2021-09-27 11:44:41,010 DEBUG awx.main.dispatch task 198b9894-11da-4261-a65e-a3e22e777b05 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:44:41,012 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:44:41,045 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:44:41,106 DEBUG awx.main.dispatch task 198b9894-11da-4261-a65e-a3e22e777b05 is finished
2021-09-27 11:44:50,960 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d566742b-abe7-4c67-a5dc-142c579a9501, queue=awx_private_queue)
[2021-09-27 11:44:50,960: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d566742b-abe7-4c67-a5dc-142c579a9501, queue=awx_private_queue)
2021-09-27 11:44:50,978 DEBUG awx.main.dispatch delivered d566742b-abe7-4c67-a5dc-142c579a9501 to worker[182] qsize 0
2021-09-27 11:44:50,980 DEBUG awx.main.dispatch task d566742b-abe7-4c67-a5dc-142c579a9501 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:44:51,024 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:44:51,030 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:44:21.076890+00:00
2021-09-27 11:44:51,117 DEBUG awx.main.dispatch task d566742b-abe7-4c67-a5dc-142c579a9501 is finished
2021-09-27 11:45:00,991 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4bad9e5d-eaed-4e9f-a1b0-b935a9e74e14, queue=awx_private_queue)
[2021-09-27 11:45:00,991: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4bad9e5d-eaed-4e9f-a1b0-b935a9e74e14, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 11:45:01,010 DEBUG awx.main.dispatch delivered 4bad9e5d-eaed-4e9f-a1b0-b935a9e74e14 to worker[180] qsize 0
2021-09-27 11:45:01,013 DEBUG awx.main.dispatch task 4bad9e5d-eaed-4e9f-a1b0-b935a9e74e14 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:45:01,015 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:45:01,049 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:45:01,113 DEBUG awx.main.dispatch task 4bad9e5d-eaed-4e9f-a1b0-b935a9e74e14 is finished
2021-09-27 11:45:20,607 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(06b40df0-670f-4f46-9e25-bf1a93bc03e7, queue=dev-awx-0)
[2021-09-27 11:45:20,607: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(06b40df0-670f-4f46-9e25-bf1a93bc03e7, queue=dev-awx-0)
2021-09-27 11:45:20,944 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a4ed8069-4283-42c7-bcd5-7b694736a4fe, queue=dev-awx-0)
[2021-09-27 11:45:20,944: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a4ed8069-4283-42c7-bcd5-7b694736a4fe, queue=dev-awx-0)
2021-09-27 11:45:20,962 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(31b763d4-bc5b-47e8-802e-d445d05bb5c2, queue=awx_private_queue)
[2021-09-27 11:45:20,962: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(31b763d4-bc5b-47e8-802e-d445d05bb5c2, queue=awx_private_queue)
2021-09-27 11:45:20,991 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(31134b88-2f43-4cf4-b9c2-1efa68c05638, queue=awx_private_queue)
[2021-09-27 11:45:20,991: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(31134b88-2f43-4cf4-b9c2-1efa68c05638, queue=awx_private_queue)
2021-09-27 11:45:21,978 DEBUG awx.main.dispatch delivered 06b40df0-670f-4f46-9e25-bf1a93bc03e7 to worker[180] qsize 0
2021-09-27 11:45:21,982 DEBUG awx.main.dispatch delivered a4ed8069-4283-42c7-bcd5-7b694736a4fe to worker[182] qsize 0
2021-09-27 11:45:21,982 DEBUG awx.main.dispatch task 06b40df0-670f-4f46-9e25-bf1a93bc03e7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:45:21,985 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:45:21,984 DEBUG awx.main.dispatch delivered 31b763d4-bc5b-47e8-802e-d445d05bb5c2 to worker[183] qsize 0
2021-09-27 11:45:21,984 DEBUG awx.main.dispatch task a4ed8069-4283-42c7-bcd5-7b694736a4fe starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:45:21,986 DEBUG awx.main.dispatch task 31b763d4-bc5b-47e8-802e-d445d05bb5c2 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:45:21,987 DEBUG awx.main.dispatch delivered 31134b88-2f43-4cf4-b9c2-1efa68c05638 to worker[181] qsize 0
2021-09-27 11:45:21,990 DEBUG awx.main.dispatch task 31134b88-2f43-4cf4-b9c2-1efa68c05638 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:45:21,992 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:45:24,152 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:45:24,158 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:44:51.024836+00:00
2021-09-27 11:45:24,162 DEBUG awx.main.dispatch task a4ed8069-4283-42c7-bcd5-7b694736a4fe is finished
2021-09-27 11:45:24,183 DEBUG awx.main.dispatch task 31b763d4-bc5b-47e8-802e-d445d05bb5c2 is finished
2021-09-27 11:45:24,194 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:45:24,234 DEBUG awx.main.dispatch task 06b40df0-670f-4f46-9e25-bf1a93bc03e7 is finished
2021-09-27 11:45:24,394 DEBUG awx.main.dispatch task 31134b88-2f43-4cf4-b9c2-1efa68c05638 is finished
2021-09-27 11:45:41,000 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(285cc91a-c7d4-47e8-84b7-9f9f86776faa, queue=awx_private_queue)
[2021-09-27 11:45:41,000: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(285cc91a-c7d4-47e8-84b7-9f9f86776faa, queue=awx_private_queue)
2021-09-27 11:45:41,022 DEBUG awx.main.dispatch delivered 285cc91a-c7d4-47e8-84b7-9f9f86776faa to worker[183] qsize 0
2021-09-27 11:45:41,026 DEBUG awx.main.dispatch task 285cc91a-c7d4-47e8-84b7-9f9f86776faa starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:45:41,027 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:45:41,043 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:45:41,073 DEBUG awx.main.dispatch task 285cc91a-c7d4-47e8-84b7-9f9f86776faa is finished
2021-09-27 11:45:50,965 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e28349be-2123-41ef-9588-afe8b7504f63, queue=awx_private_queue)
[2021-09-27 11:45:50,965: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e28349be-2123-41ef-9588-afe8b7504f63, queue=awx_private_queue)
2021-09-27 11:45:50,991 DEBUG awx.main.dispatch delivered e28349be-2123-41ef-9588-afe8b7504f63 to worker[180] qsize 0
2021-09-27 11:45:50,994 DEBUG awx.main.dispatch task e28349be-2123-41ef-9588-afe8b7504f63 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:45:51,023 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:45:51,027 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:45:24.153141+00:00
2021-09-27 11:45:51,047 DEBUG awx.main.dispatch task e28349be-2123-41ef-9588-afe8b7504f63 is finished
2021-09-27 11:46:01,001 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(abb50935-da4b-4f10-82ab-9598427b6cfd, queue=awx_private_queue)
[2021-09-27 11:46:01,001: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(abb50935-da4b-4f10-82ab-9598427b6cfd, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 11:46:01,024 DEBUG awx.main.dispatch delivered abb50935-da4b-4f10-82ab-9598427b6cfd to worker[180] qsize 0
2021-09-27 11:46:01,027 DEBUG awx.main.dispatch task abb50935-da4b-4f10-82ab-9598427b6cfd starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:46:01,028 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:46:01,057 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:46:01,084 DEBUG awx.main.dispatch task abb50935-da4b-4f10-82ab-9598427b6cfd is finished
2021-09-27 11:46:20,617 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1b32e664-28f8-48d5-b346-f2844ac88edb, queue=dev-awx-0)
[2021-09-27 11:46:20,617: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1b32e664-28f8-48d5-b346-f2844ac88edb, queue=dev-awx-0)
2021-09-27 11:46:20,943 DEBUG awx.main.dispatch delivered 1b32e664-28f8-48d5-b346-f2844ac88edb to worker[180] qsize 0
2021-09-27 11:46:20,944 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(f79ce80b-4dc5-4261-aa2b-8f2243a8d94d, queue=dev-awx-0)
[2021-09-27 11:46:20,944: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(f79ce80b-4dc5-4261-aa2b-8f2243a8d94d, queue=dev-awx-0)
2021-09-27 11:46:20,946 DEBUG awx.main.dispatch task 1b32e664-28f8-48d5-b346-f2844ac88edb starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:46:20,948 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:46:20,962 DEBUG awx.main.dispatch delivered f79ce80b-4dc5-4261-aa2b-8f2243a8d94d to worker[182] qsize 0
2021-09-27 11:46:20,964 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(02cd9e6c-b4be-45bf-920f-a04b616b10ed, queue=awx_private_queue)
[2021-09-27 11:46:20,964: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(02cd9e6c-b4be-45bf-920f-a04b616b10ed, queue=awx_private_queue)
2021-09-27 11:46:20,965 DEBUG awx.main.dispatch task f79ce80b-4dc5-4261-aa2b-8f2243a8d94d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:46:20,986 DEBUG awx.main.dispatch delivered 02cd9e6c-b4be-45bf-920f-a04b616b10ed to worker[183] qsize 0
2021-09-27 11:46:20,989 DEBUG awx.main.dispatch task 02cd9e6c-b4be-45bf-920f-a04b616b10ed starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:46:21,000 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(661e6eb0-c682-4989-af06-274a8c08a033, queue=awx_private_queue)
[2021-09-27 11:46:21,000: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(661e6eb0-c682-4989-af06-274a8c08a033, queue=awx_private_queue)
2021-09-27 11:46:21,007 DEBUG awx.main.dispatch task f79ce80b-4dc5-4261-aa2b-8f2243a8d94d is finished
2021-09-27 11:46:21,019 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:46:21,024 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:45:51.024422+00:00
2021-09-27 11:46:21,023 DEBUG awx.main.dispatch delivered 661e6eb0-c682-4989-af06-274a8c08a033 to worker[181] qsize 0
2021-09-27 11:46:21,026 DEBUG awx.main.dispatch task 661e6eb0-c682-4989-af06-274a8c08a033 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:46:21,028 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:46:21,033 DEBUG awx.main.dispatch task 1b32e664-28f8-48d5-b346-f2844ac88edb is finished
2021-09-27 11:46:21,055 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:46:21,055 DEBUG awx.main.dispatch task 02cd9e6c-b4be-45bf-920f-a04b616b10ed is finished
2021-09-27 11:46:21,135 DEBUG awx.main.dispatch task 661e6eb0-c682-4989-af06-274a8c08a033 is finished
2021-09-27 11:46:41,014 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(01a6c53e-fe0c-446f-aa39-bcc7e5675f95, queue=awx_private_queue)
[2021-09-27 11:46:41,014: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(01a6c53e-fe0c-446f-aa39-bcc7e5675f95, queue=awx_private_queue)
2021-09-27 11:46:41,038 DEBUG awx.main.dispatch delivered 01a6c53e-fe0c-446f-aa39-bcc7e5675f95 to worker[181] qsize 0
2021-09-27 11:46:41,042 DEBUG awx.main.dispatch task 01a6c53e-fe0c-446f-aa39-bcc7e5675f95 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:46:41,045 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:46:41,762 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:46:41,975 DEBUG awx.main.dispatch task 01a6c53e-fe0c-446f-aa39-bcc7e5675f95 is finished
2021-09-27 11:46:50,964 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(12d3904f-cfee-4788-9ba6-813e24ede989, queue=awx_private_queue)
[2021-09-27 11:46:50,964: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(12d3904f-cfee-4788-9ba6-813e24ede989, queue=awx_private_queue)
2021-09-27 11:46:50,990 DEBUG awx.main.dispatch delivered 12d3904f-cfee-4788-9ba6-813e24ede989 to worker[183] qsize 0
2021-09-27 11:46:50,993 DEBUG awx.main.dispatch task 12d3904f-cfee-4788-9ba6-813e24ede989 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:46:51,022 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:46:51,031 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:46:21.020134+00:00
2021-09-27 11:46:51,055 DEBUG awx.main.dispatch task 12d3904f-cfee-4788-9ba6-813e24ede989 is finished
2021-09-27 11:47:01,014 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(67920c29-624d-4cff-b4d2-e17f49b9e7fe, queue=awx_private_queue)
[2021-09-27 11:47:01,014: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(67920c29-624d-4cff-b4d2-e17f49b9e7fe, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 11:47:01,038 DEBUG awx.main.dispatch delivered 67920c29-624d-4cff-b4d2-e17f49b9e7fe to worker[180] qsize 0
2021-09-27 11:47:01,041 DEBUG awx.main.dispatch task 67920c29-624d-4cff-b4d2-e17f49b9e7fe starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:47:01,044 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:47:01,069 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:47:01,129 DEBUG awx.main.dispatch task 67920c29-624d-4cff-b4d2-e17f49b9e7fe is finished
2021-09-27 11:47:20,627 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(d46b2fd3-f45f-4c12-994d-b041f4223067, queue=dev-awx-0)
[2021-09-27 11:47:20,627: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(d46b2fd3-f45f-4c12-994d-b041f4223067, queue=dev-awx-0)
2021-09-27 11:47:20,933 DEBUG awx.main.dispatch delivered d46b2fd3-f45f-4c12-994d-b041f4223067 to worker[182] qsize 0
2021-09-27 11:47:20,935 DEBUG awx.main.dispatch task d46b2fd3-f45f-4c12-994d-b041f4223067 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:47:20,937 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:47:20,944 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(73e5ab20-6ef2-4e76-b01f-9ec0a965eb30, queue=dev-awx-0)
[2021-09-27 11:47:20,944: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(73e5ab20-6ef2-4e76-b01f-9ec0a965eb30, queue=dev-awx-0)
2021-09-27 11:47:20,960 DEBUG awx.main.dispatch delivered 73e5ab20-6ef2-4e76-b01f-9ec0a965eb30 to worker[183] qsize 0
2021-09-27 11:47:20,962 DEBUG awx.main.dispatch task 73e5ab20-6ef2-4e76-b01f-9ec0a965eb30 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:47:20,964 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8dd620d8-298e-40ec-b514-dcc0a967346d, queue=awx_private_queue)
[2021-09-27 11:47:20,964: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8dd620d8-298e-40ec-b514-dcc0a967346d, queue=awx_private_queue)
2021-09-27 11:47:20,979 DEBUG awx.main.dispatch task d46b2fd3-f45f-4c12-994d-b041f4223067 is finished
2021-09-27 11:47:20,979 DEBUG awx.main.dispatch delivered 8dd620d8-298e-40ec-b514-dcc0a967346d to worker[180] qsize 0
2021-09-27 11:47:20,982 DEBUG awx.main.dispatch task 8dd620d8-298e-40ec-b514-dcc0a967346d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:47:21,014 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(541d78d0-9fab-48d4-b7e8-ff38d7a449da, queue=awx_private_queue)
[2021-09-27 11:47:21,014: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(541d78d0-9fab-48d4-b7e8-ff38d7a449da, queue=awx_private_queue)
2021-09-27 11:47:21,015 DEBUG awx.main.dispatch task 73e5ab20-6ef2-4e76-b01f-9ec0a965eb30 is finished
2021-09-27 11:47:21,022 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:47:21,025 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:46:51.022565+00:00
2021-09-27 11:47:21,037 DEBUG awx.main.dispatch delivered 541d78d0-9fab-48d4-b7e8-ff38d7a449da to worker[182] qsize 0
2021-09-27 11:47:21,041 DEBUG awx.main.dispatch task 541d78d0-9fab-48d4-b7e8-ff38d7a449da starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:47:21,043 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:47:21,049 DEBUG awx.main.dispatch task 8dd620d8-298e-40ec-b514-dcc0a967346d is finished
2021-09-27 11:47:21,070 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:47:21,135 DEBUG awx.main.dispatch task 541d78d0-9fab-48d4-b7e8-ff38d7a449da is finished
2021-09-27 11:47:41,024 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2e47fb7a-1b65-4251-a21d-d62c2227b555, queue=awx_private_queue)
[2021-09-27 11:47:41,024: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2e47fb7a-1b65-4251-a21d-d62c2227b555, queue=awx_private_queue)
2021-09-27 11:47:41,043 DEBUG awx.main.dispatch delivered 2e47fb7a-1b65-4251-a21d-d62c2227b555 to worker[183] qsize 0
2021-09-27 11:47:41,047 DEBUG awx.main.dispatch task 2e47fb7a-1b65-4251-a21d-d62c2227b555 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:47:41,049 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:47:42,171 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:47:42,285 DEBUG awx.main.dispatch task 2e47fb7a-1b65-4251-a21d-d62c2227b555 is finished
2021-09-27 11:47:50,965 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e0633582-5518-4491-b759-b149933d55b9, queue=awx_private_queue)
[2021-09-27 11:47:50,965: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e0633582-5518-4491-b759-b149933d55b9, queue=awx_private_queue)
2021-09-27 11:47:50,984 DEBUG awx.main.dispatch delivered e0633582-5518-4491-b759-b149933d55b9 to worker[181] qsize 0
2021-09-27 11:47:50,988 DEBUG awx.main.dispatch task e0633582-5518-4491-b759-b149933d55b9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:47:58,280 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:47:58,297 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:47:21.022738+00:00
2021-09-27 11:47:58,334 DEBUG awx.main.dispatch task e0633582-5518-4491-b759-b149933d55b9 is finished
2021-09-27 11:48:01,023 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cdfe7262-0c44-4887-bf49-e00712cfbff8, queue=awx_private_queue)
[2021-09-27 11:48:01,023: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cdfe7262-0c44-4887-bf49-e00712cfbff8, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 11:48:01,062 DEBUG awx.main.dispatch delivered cdfe7262-0c44-4887-bf49-e00712cfbff8 to worker[182] qsize 0
2021-09-27 11:48:01,065 DEBUG awx.main.dispatch task cdfe7262-0c44-4887-bf49-e00712cfbff8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:48:01,067 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:48:02,307 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:48:02,358 DEBUG awx.main.dispatch task cdfe7262-0c44-4887-bf49-e00712cfbff8 is finished
2021-09-27 11:48:20,637 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6fb9b527-dd83-4392-83a9-629e42e84055, queue=dev-awx-0)
[2021-09-27 11:48:20,637: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6fb9b527-dd83-4392-83a9-629e42e84055, queue=dev-awx-0)
2021-09-27 11:48:20,985 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c1d6133f-db7a-43d7-b0e2-5657ae3d2868, queue=dev-awx-0)
[2021-09-27 11:48:20,985: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c1d6133f-db7a-43d7-b0e2-5657ae3d2868, queue=dev-awx-0)
2021-09-27 11:48:21,003 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aa7a49b5-1efd-45ab-b543-143c26898f21, queue=awx_private_queue)
[2021-09-27 11:48:21,003: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aa7a49b5-1efd-45ab-b543-143c26898f21, queue=awx_private_queue)
2021-09-27 11:48:21,022 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a6d5a0b1-2881-483f-8f5c-2c6ebae25004, queue=awx_private_queue)
[2021-09-27 11:48:21,022: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a6d5a0b1-2881-483f-8f5c-2c6ebae25004, queue=awx_private_queue)
2021-09-27 11:48:21,416 DEBUG awx.main.dispatch delivered 6fb9b527-dd83-4392-83a9-629e42e84055 to worker[183] qsize 0
2021-09-27 11:48:21,421 DEBUG awx.main.dispatch delivered c1d6133f-db7a-43d7-b0e2-5657ae3d2868 to worker[180] qsize 0
2021-09-27 11:48:21,422 DEBUG awx.main.dispatch task 6fb9b527-dd83-4392-83a9-629e42e84055 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:48:21,426 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:48:21,424 DEBUG awx.main.dispatch delivered aa7a49b5-1efd-45ab-b543-143c26898f21 to worker[182] qsize 0
2021-09-27 11:48:21,427 DEBUG awx.main.dispatch delivered a6d5a0b1-2881-483f-8f5c-2c6ebae25004 to worker[181] qsize 0
2021-09-27 11:48:21,425 DEBUG awx.main.dispatch task c1d6133f-db7a-43d7-b0e2-5657ae3d2868 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:48:21,427 DEBUG awx.main.dispatch task aa7a49b5-1efd-45ab-b543-143c26898f21 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:48:21,430 DEBUG awx.main.dispatch task a6d5a0b1-2881-483f-8f5c-2c6ebae25004 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:48:21,432 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:48:21,478 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:48:21,502 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:48:21,515 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:47:58.281381+00:00
2021-09-27 11:48:21,516 DEBUG awx.main.dispatch task c1d6133f-db7a-43d7-b0e2-5657ae3d2868 is finished
2021-09-27 11:48:21,549 DEBUG awx.main.dispatch task aa7a49b5-1efd-45ab-b543-143c26898f21 is finished
2021-09-27 11:48:21,550 DEBUG awx.main.dispatch task a6d5a0b1-2881-483f-8f5c-2c6ebae25004 is finished
2021-09-27 11:48:21,553 DEBUG awx.main.dispatch task 6fb9b527-dd83-4392-83a9-629e42e84055 is finished
2021-09-27 11:48:41,036 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bf024198-acbc-4d4e-9fc2-899ec0d36cd7, queue=awx_private_queue)
[2021-09-27 11:48:41,036: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bf024198-acbc-4d4e-9fc2-899ec0d36cd7, queue=awx_private_queue)
2021-09-27 11:48:41,064 DEBUG awx.main.dispatch delivered bf024198-acbc-4d4e-9fc2-899ec0d36cd7 to worker[182] qsize 0
2021-09-27 11:48:41,070 DEBUG awx.main.dispatch task bf024198-acbc-4d4e-9fc2-899ec0d36cd7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:48:41,072 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:48:50,839 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:48:51,004 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(a1b2fdcf-934e-4f45-8410-d0105cb268cc, queue=awx_private_queue)
[2021-09-27 11:48:51,004: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(a1b2fdcf-934e-4f45-8410-d0105cb268cc, queue=awx_private_queue)
2021-09-27 11:48:51,027 DEBUG awx.main.dispatch delivered a1b2fdcf-934e-4f45-8410-d0105cb268cc to worker[181] qsize 0
2021-09-27 11:48:51,030 DEBUG awx.main.dispatch task a1b2fdcf-934e-4f45-8410-d0105cb268cc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:48:52,684 DEBUG awx.main.dispatch task bf024198-acbc-4d4e-9fc2-899ec0d36cd7 is finished
2021-09-27 11:48:53,625 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:48:53,727 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:48:21.479253+00:00
2021-09-27 11:48:53,859 DEBUG awx.main.dispatch task a1b2fdcf-934e-4f45-8410-d0105cb268cc is finished
2021-09-27 11:49:01,035 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7bfcf43e-f66c-4fbd-a98b-6b07ff349f82, queue=awx_private_queue)
[2021-09-27 11:49:01,035: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7bfcf43e-f66c-4fbd-a98b-6b07ff349f82, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 11:49:01,059 DEBUG awx.main.dispatch delivered 7bfcf43e-f66c-4fbd-a98b-6b07ff349f82 to worker[180] qsize 0
2021-09-27 11:49:01,062 DEBUG awx.main.dispatch task 7bfcf43e-f66c-4fbd-a98b-6b07ff349f82 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:49:01,065 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:49:02,097 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:49:02,103 DEBUG awx.main.dispatch task 7bfcf43e-f66c-4fbd-a98b-6b07ff349f82 is finished
2021-09-27 11:49:20,643 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(21942d12-7e9e-44a8-ad56-272e47d33a23, queue=dev-awx-0)
[2021-09-27 11:49:20,643: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(21942d12-7e9e-44a8-ad56-272e47d33a23, queue=dev-awx-0)
2021-09-27 11:49:21,965 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3fbb8376-efcb-4cb4-8439-ff56a9057f55, queue=dev-awx-0)
[2021-09-27 11:49:21,965: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3fbb8376-efcb-4cb4-8439-ff56a9057f55, queue=dev-awx-0)
2021-09-27 11:49:21,991 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7446ca9e-0a5d-437c-a9bd-7c2318758bb1, queue=awx_private_queue)
[2021-09-27 11:49:21,991: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7446ca9e-0a5d-437c-a9bd-7c2318758bb1, queue=awx_private_queue)
2021-09-27 11:49:22,018 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b42e83bc-7bf0-443e-8cce-025a750bada8, queue=awx_private_queue)
[2021-09-27 11:49:22,018: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b42e83bc-7bf0-443e-8cce-025a750bada8, queue=awx_private_queue)
2021-09-27 11:49:22,026 DEBUG awx.main.dispatch delivered 21942d12-7e9e-44a8-ad56-272e47d33a23 to worker[182] qsize 0
2021-09-27 11:49:22,029 DEBUG awx.main.dispatch delivered 3fbb8376-efcb-4cb4-8439-ff56a9057f55 to worker[183] qsize 0
2021-09-27 11:49:22,029 DEBUG awx.main.dispatch task 21942d12-7e9e-44a8-ad56-272e47d33a23 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:49:22,032 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:49:22,031 DEBUG awx.main.dispatch delivered 7446ca9e-0a5d-437c-a9bd-7c2318758bb1 to worker[180] qsize 0
2021-09-27 11:49:22,036 DEBUG awx.main.dispatch task 7446ca9e-0a5d-437c-a9bd-7c2318758bb1 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:49:22,037 DEBUG awx.main.dispatch task 3fbb8376-efcb-4cb4-8439-ff56a9057f55 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:49:22,045 DEBUG awx.main.dispatch delivered b42e83bc-7bf0-443e-8cce-025a750bada8 to worker[181] qsize 0
2021-09-27 11:49:22,049 DEBUG awx.main.dispatch task b42e83bc-7bf0-443e-8cce-025a750bada8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:49:22,051 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:49:22,171 DEBUG awx.main.dispatch task 21942d12-7e9e-44a8-ad56-272e47d33a23 is finished
2021-09-27 11:49:22,213 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:49:22,234 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:48:53.626215+00:00
2021-09-27 11:49:22,236 DEBUG awx.main.dispatch task 3fbb8376-efcb-4cb4-8439-ff56a9057f55 is finished
2021-09-27 11:49:22,418 DEBUG awx.main.dispatch task 7446ca9e-0a5d-437c-a9bd-7c2318758bb1 is finished
2021-09-27 11:49:22,524 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:49:22,621 DEBUG awx.main.dispatch task b42e83bc-7bf0-443e-8cce-025a750bada8 is finished
2021-09-27 11:49:42,034 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bb5a5422-fbce-4de2-a1f6-26420d5ec8f8, queue=awx_private_queue)
[2021-09-27 11:49:42,034: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bb5a5422-fbce-4de2-a1f6-26420d5ec8f8, queue=awx_private_queue)
2021-09-27 11:49:42,065 DEBUG awx.main.dispatch delivered bb5a5422-fbce-4de2-a1f6-26420d5ec8f8 to worker[182] qsize 0
2021-09-27 11:49:42,069 DEBUG awx.main.dispatch task bb5a5422-fbce-4de2-a1f6-26420d5ec8f8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:49:42,071 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:49:42,110 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:49:42,164 DEBUG awx.main.dispatch task bb5a5422-fbce-4de2-a1f6-26420d5ec8f8 is finished
2021-09-27 11:49:51,992 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e91b85c5-2af4-4ae6-96e7-5311279f5810, queue=awx_private_queue)
[2021-09-27 11:49:51,992: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e91b85c5-2af4-4ae6-96e7-5311279f5810, queue=awx_private_queue)
2021-09-27 11:49:52,019 DEBUG awx.main.dispatch delivered e91b85c5-2af4-4ae6-96e7-5311279f5810 to worker[182] qsize 0
2021-09-27 11:49:52,022 DEBUG awx.main.dispatch task e91b85c5-2af4-4ae6-96e7-5311279f5810 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:49:52,038 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:49:52,042 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:49:22.213985+00:00
2021-09-27 11:49:52,063 DEBUG awx.main.dispatch task e91b85c5-2af4-4ae6-96e7-5311279f5810 is finished
RESULT 2
OKREADY
2021-09-27 11:50:02,034 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(344ca247-1013-4b92-aac1-3a0be87bd9e8, queue=awx_private_queue)
[2021-09-27 11:50:02,034: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(344ca247-1013-4b92-aac1-3a0be87bd9e8, queue=awx_private_queue)
2021-09-27 11:50:02,056 DEBUG awx.main.dispatch delivered 344ca247-1013-4b92-aac1-3a0be87bd9e8 to worker[183] qsize 0
2021-09-27 11:50:02,059 DEBUG awx.main.dispatch task 344ca247-1013-4b92-aac1-3a0be87bd9e8 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:50:02,062 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:50:02,152 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:50:02,258 DEBUG awx.main.dispatch task 344ca247-1013-4b92-aac1-3a0be87bd9e8 is finished
2021-09-27 11:50:20,655 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4bf8a44d-fe47-4892-a9aa-1dcab070ee8a, queue=dev-awx-0)
[2021-09-27 11:50:20,655: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4bf8a44d-fe47-4892-a9aa-1dcab070ee8a, queue=dev-awx-0)
2021-09-27 11:50:20,741 DEBUG awx.main.dispatch delivered 4bf8a44d-fe47-4892-a9aa-1dcab070ee8a to worker[180] qsize 0
2021-09-27 11:50:20,745 DEBUG awx.main.dispatch task 4bf8a44d-fe47-4892-a9aa-1dcab070ee8a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:50:20,748 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:50:20,846 DEBUG awx.main.dispatch task 4bf8a44d-fe47-4892-a9aa-1dcab070ee8a is finished
2021-09-27 11:50:21,966 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(16d0a6c9-6959-4a0e-828a-bb1e487debb4, queue=dev-awx-0)
[2021-09-27 11:50:21,966: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(16d0a6c9-6959-4a0e-828a-bb1e487debb4, queue=dev-awx-0)
2021-09-27 11:50:23,367 DEBUG awx.main.dispatch delivered 16d0a6c9-6959-4a0e-828a-bb1e487debb4 to worker[183] qsize 0
2021-09-27 11:50:23,369 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(248980b6-3b58-4ff6-a09e-09d2c1cab07a, queue=awx_private_queue)
[2021-09-27 11:50:23,369: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(248980b6-3b58-4ff6-a09e-09d2c1cab07a, queue=awx_private_queue)
2021-09-27 11:50:23,390 DEBUG awx.main.dispatch delivered 248980b6-3b58-4ff6-a09e-09d2c1cab07a to worker[181] qsize 0
2021-09-27 11:50:23,392 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(64cbc28e-431c-46a9-8c9b-9dd4406e8053, queue=awx_private_queue)
[2021-09-27 11:50:23,392: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(64cbc28e-431c-46a9-8c9b-9dd4406e8053, queue=awx_private_queue)
2021-09-27 11:50:23,411 DEBUG awx.main.dispatch delivered 64cbc28e-431c-46a9-8c9b-9dd4406e8053 to worker[182] qsize 0
2021-09-27 11:50:23,370 DEBUG awx.main.dispatch task 16d0a6c9-6959-4a0e-828a-bb1e487debb4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:50:23,393 DEBUG awx.main.dispatch task 248980b6-3b58-4ff6-a09e-09d2c1cab07a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:50:23,413 DEBUG awx.main.dispatch task 64cbc28e-431c-46a9-8c9b-9dd4406e8053 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:50:25,497 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:50:25,533 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:50:25,536 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:49:52.038428+00:00
2021-09-27 11:50:25,574 DEBUG awx.main.dispatch task 248980b6-3b58-4ff6-a09e-09d2c1cab07a is finished
2021-09-27 11:50:25,712 DEBUG awx.main.dispatch task 16d0a6c9-6959-4a0e-828a-bb1e487debb4 is finished
2021-09-27 11:50:25,737 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:50:26,072 DEBUG awx.main.dispatch task 64cbc28e-431c-46a9-8c9b-9dd4406e8053 is finished
2021-09-27 11:50:43,402 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f7c9c7e2-6be6-4177-b336-3b51e31e6a98, queue=awx_private_queue)
[2021-09-27 11:50:43,402: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f7c9c7e2-6be6-4177-b336-3b51e31e6a98, queue=awx_private_queue)
2021-09-27 11:50:43,422 DEBUG awx.main.dispatch delivered f7c9c7e2-6be6-4177-b336-3b51e31e6a98 to worker[180] qsize 0
2021-09-27 11:50:43,426 DEBUG awx.main.dispatch task f7c9c7e2-6be6-4177-b336-3b51e31e6a98 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:50:43,428 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:50:44,433 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:50:44,436 DEBUG awx.main.dispatch task f7c9c7e2-6be6-4177-b336-3b51e31e6a98 is finished
2021-09-27 11:50:53,368 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ee21e2ee-21c5-41c9-84be-580c4a8f88e7, queue=awx_private_queue)
[2021-09-27 11:50:53,368: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ee21e2ee-21c5-41c9-84be-580c4a8f88e7, queue=awx_private_queue)
2021-09-27 11:50:53,391 DEBUG awx.main.dispatch delivered ee21e2ee-21c5-41c9-84be-580c4a8f88e7 to worker[182] qsize 0
2021-09-27 11:50:53,394 DEBUG awx.main.dispatch task ee21e2ee-21c5-41c9-84be-580c4a8f88e7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:50:54,417 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:50:54,423 DEBUG awx.main.dispatch task ee21e2ee-21c5-41c9-84be-580c4a8f88e7 is finished
RESULT 2
OKREADY
2021-09-27 11:51:03,403 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ab8040de-1e67-455a-b857-56fcfb44ea03, queue=awx_private_queue)
[2021-09-27 11:51:03,403: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ab8040de-1e67-455a-b857-56fcfb44ea03, queue=awx_private_queue)
2021-09-27 11:51:03,429 DEBUG awx.main.dispatch delivered ab8040de-1e67-455a-b857-56fcfb44ea03 to worker[182] qsize 0
2021-09-27 11:51:03,432 DEBUG awx.main.dispatch task ab8040de-1e67-455a-b857-56fcfb44ea03 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:51:03,434 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:51:05,478 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:51:05,558 DEBUG awx.main.dispatch task ab8040de-1e67-455a-b857-56fcfb44ea03 is finished
2021-09-27 11:51:20,653 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(f9d39811-344e-4f5e-8aee-4815229fb724, queue=dev-awx-0)
[2021-09-27 11:51:20,653: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(f9d39811-344e-4f5e-8aee-4815229fb724, queue=dev-awx-0)
2021-09-27 11:51:21,077 DEBUG awx.main.dispatch delivered f9d39811-344e-4f5e-8aee-4815229fb724 to worker[182] qsize 0
2021-09-27 11:51:21,080 DEBUG awx.main.dispatch task f9d39811-344e-4f5e-8aee-4815229fb724 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:51:21,083 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:51:21,966 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1832a34b-c304-4959-a517-70c8a6a0631f, queue=dev-awx-0)
[2021-09-27 11:51:21,966: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1832a34b-c304-4959-a517-70c8a6a0631f, queue=dev-awx-0)
2021-09-27 11:51:21,991 DEBUG awx.main.dispatch delivered 1832a34b-c304-4959-a517-70c8a6a0631f to worker[183] qsize 0
2021-09-27 11:51:21,994 DEBUG awx.main.dispatch task 1832a34b-c304-4959-a517-70c8a6a0631f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:51:22,282 DEBUG awx.main.dispatch task 1832a34b-c304-4959-a517-70c8a6a0631f is finished
2021-09-27 11:51:22,356 DEBUG awx.main.dispatch task f9d39811-344e-4f5e-8aee-4815229fb724 is finished
2021-09-27 11:51:23,369 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(80a82e1b-75e0-4228-8e4a-f0cb7e7e7280, queue=awx_private_queue)
[2021-09-27 11:51:23,369: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(80a82e1b-75e0-4228-8e4a-f0cb7e7e7280, queue=awx_private_queue)
2021-09-27 11:51:23,401 DEBUG awx.main.dispatch delivered 80a82e1b-75e0-4228-8e4a-f0cb7e7e7280 to worker[183] qsize 0
2021-09-27 11:51:23,402 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(617caaff-9864-4d83-9961-263bcb8e0cae, queue=awx_private_queue)
[2021-09-27 11:51:23,402: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(617caaff-9864-4d83-9961-263bcb8e0cae, queue=awx_private_queue)
2021-09-27 11:51:23,403 DEBUG awx.main.dispatch task 80a82e1b-75e0-4228-8e4a-f0cb7e7e7280 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:51:23,426 DEBUG awx.main.dispatch delivered 617caaff-9864-4d83-9961-263bcb8e0cae to worker[180] qsize 0
2021-09-27 11:51:23,429 DEBUG awx.main.dispatch task 617caaff-9864-4d83-9961-263bcb8e0cae starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:51:23,431 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:51:23,443 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:51:23,447 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:50:25.533702+00:00
2021-09-27 11:51:23,465 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:51:23,499 DEBUG awx.main.dispatch task 80a82e1b-75e0-4228-8e4a-f0cb7e7e7280 is finished
2021-09-27 11:51:23,569 DEBUG awx.main.dispatch task 617caaff-9864-4d83-9961-263bcb8e0cae is finished
2021-09-27 11:51:43,403 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7e2d823f-c2d0-46b6-915d-03d76e4ec885, queue=awx_private_queue)
[2021-09-27 11:51:43,403: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7e2d823f-c2d0-46b6-915d-03d76e4ec885, queue=awx_private_queue)
2021-09-27 11:51:45,097 DEBUG awx.main.dispatch delivered 7e2d823f-c2d0-46b6-915d-03d76e4ec885 to worker[181] qsize 0
2021-09-27 11:51:45,391 DEBUG awx.main.dispatch task 7e2d823f-c2d0-46b6-915d-03d76e4ec885 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:51:45,395 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:51:46,776 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:51:47,570 DEBUG awx.main.dispatch task 7e2d823f-c2d0-46b6-915d-03d76e4ec885 is finished
2021-09-27 11:51:53,369 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ac2dda62-1693-4ce9-ac60-5cb78c72161c, queue=awx_private_queue)
[2021-09-27 11:51:53,369: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ac2dda62-1693-4ce9-ac60-5cb78c72161c, queue=awx_private_queue)
2021-09-27 11:51:53,392 DEBUG awx.main.dispatch delivered ac2dda62-1693-4ce9-ac60-5cb78c72161c to worker[182] qsize 0
2021-09-27 11:51:53,394 DEBUG awx.main.dispatch task ac2dda62-1693-4ce9-ac60-5cb78c72161c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:51:54,417 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:51:54,421 DEBUG awx.main.dispatch task ac2dda62-1693-4ce9-ac60-5cb78c72161c is finished
RESULT 2
OKREADY
2021-09-27 11:52:03,406 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ac02290f-d8bf-4fe6-bcf4-926b3b4d6718, queue=awx_private_queue)
[2021-09-27 11:52:03,406: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ac02290f-d8bf-4fe6-bcf4-926b3b4d6718, queue=awx_private_queue)
2021-09-27 11:52:03,438 DEBUG awx.main.dispatch delivered ac02290f-d8bf-4fe6-bcf4-926b3b4d6718 to worker[182] qsize 0
2021-09-27 11:52:03,442 DEBUG awx.main.dispatch task ac02290f-d8bf-4fe6-bcf4-926b3b4d6718 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:52:03,445 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:52:04,465 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:52:04,473 DEBUG awx.main.dispatch task ac02290f-d8bf-4fe6-bcf4-926b3b4d6718 is finished
2021-09-27 11:52:20,653 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e7d44ac6-bd32-4d7f-91b2-c9977431e2d9, queue=dev-awx-0)
[2021-09-27 11:52:20,653: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e7d44ac6-bd32-4d7f-91b2-c9977431e2d9, queue=dev-awx-0)
2021-09-27 11:52:20,751 DEBUG awx.main.dispatch delivered e7d44ac6-bd32-4d7f-91b2-c9977431e2d9 to worker[181] qsize 0
2021-09-27 11:52:20,755 DEBUG awx.main.dispatch task e7d44ac6-bd32-4d7f-91b2-c9977431e2d9 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:52:20,758 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:52:20,819 DEBUG awx.main.dispatch task e7d44ac6-bd32-4d7f-91b2-c9977431e2d9 is finished
2021-09-27 11:52:21,966 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d318674b-674c-4787-a5c3-1ec0ff970df1, queue=dev-awx-0)
[2021-09-27 11:52:21,966: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d318674b-674c-4787-a5c3-1ec0ff970df1, queue=dev-awx-0)
2021-09-27 11:52:21,992 DEBUG awx.main.dispatch delivered d318674b-674c-4787-a5c3-1ec0ff970df1 to worker[180] qsize 0
2021-09-27 11:52:21,996 DEBUG awx.main.dispatch task d318674b-674c-4787-a5c3-1ec0ff970df1 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:52:22,039 DEBUG awx.main.dispatch task d318674b-674c-4787-a5c3-1ec0ff970df1 is finished
2021-09-27 11:52:23,369 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0db925ee-ce3b-4613-83f1-801369cb481c, queue=awx_private_queue)
[2021-09-27 11:52:23,369: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0db925ee-ce3b-4613-83f1-801369cb481c, queue=awx_private_queue)
2021-09-27 11:52:23,399 DEBUG awx.main.dispatch delivered 0db925ee-ce3b-4613-83f1-801369cb481c to worker[181] qsize 0
2021-09-27 11:52:23,403 DEBUG awx.main.dispatch task 0db925ee-ce3b-4613-83f1-801369cb481c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:52:23,404 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(64585962-3dae-4399-aa2d-1fff6a3990b7, queue=awx_private_queue)
[2021-09-27 11:52:23,404: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(64585962-3dae-4399-aa2d-1fff6a3990b7, queue=awx_private_queue)
2021-09-27 11:52:23,422 DEBUG awx.main.dispatch delivered 64585962-3dae-4399-aa2d-1fff6a3990b7 to worker[183] qsize 0
2021-09-27 11:52:23,425 DEBUG awx.main.dispatch task 64585962-3dae-4399-aa2d-1fff6a3990b7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:52:23,429 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:52:24,377 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:52:24,382 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:51:23.443391+00:00
2021-09-27 11:52:24,415 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:52:24,431 DEBUG awx.main.dispatch task 0db925ee-ce3b-4613-83f1-801369cb481c is finished
2021-09-27 11:52:24,486 DEBUG awx.main.dispatch task 64585962-3dae-4399-aa2d-1fff6a3990b7 is finished
2021-09-27 11:52:43,404 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ea6b73e3-224c-4a7c-a139-3c1e926f1a19, queue=awx_private_queue)
[2021-09-27 11:52:43,404: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ea6b73e3-224c-4a7c-a139-3c1e926f1a19, queue=awx_private_queue)
2021-09-27 11:52:43,428 DEBUG awx.main.dispatch delivered ea6b73e3-224c-4a7c-a139-3c1e926f1a19 to worker[182] qsize 0
2021-09-27 11:52:43,431 DEBUG awx.main.dispatch task ea6b73e3-224c-4a7c-a139-3c1e926f1a19 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:52:43,434 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:52:43,454 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:52:43,486 DEBUG awx.main.dispatch task ea6b73e3-224c-4a7c-a139-3c1e926f1a19 is finished
2021-09-27 11:52:53,371 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3b6f0151-f4d0-4bd9-8502-bebf52ba05dc, queue=awx_private_queue)
[2021-09-27 11:52:53,371: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3b6f0151-f4d0-4bd9-8502-bebf52ba05dc, queue=awx_private_queue)
2021-09-27 11:52:53,583 DEBUG awx.main.dispatch delivered 3b6f0151-f4d0-4bd9-8502-bebf52ba05dc to worker[182] qsize 0
2021-09-27 11:52:53,813 DEBUG awx.main.dispatch task 3b6f0151-f4d0-4bd9-8502-bebf52ba05dc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:52:53,872 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:52:53,877 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:52:24.377592+00:00
2021-09-27 11:52:53,936 DEBUG awx.main.dispatch task 3b6f0151-f4d0-4bd9-8502-bebf52ba05dc is finished
RESULT 2
OKREADY
2021-09-27 11:53:03,405 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(de1ef880-bcee-4060-9532-047fbe2d697a, queue=awx_private_queue)
[2021-09-27 11:53:03,405: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(de1ef880-bcee-4060-9532-047fbe2d697a, queue=awx_private_queue)
2021-09-27 11:53:03,431 DEBUG awx.main.dispatch delivered de1ef880-bcee-4060-9532-047fbe2d697a to worker[181] qsize 0
2021-09-27 11:53:03,434 DEBUG awx.main.dispatch task de1ef880-bcee-4060-9532-047fbe2d697a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:53:03,438 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:53:03,969 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:53:04,054 DEBUG awx.main.dispatch task de1ef880-bcee-4060-9532-047fbe2d697a is finished
2021-09-27 11:53:20,586 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(f69224f7-1c2a-49f5-993b-5472252c82c6, queue=dev-awx-0)
[2021-09-27 11:53:20,586: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(f69224f7-1c2a-49f5-993b-5472252c82c6, queue=dev-awx-0)
2021-09-27 11:53:20,607 DEBUG awx.main.dispatch delivered f69224f7-1c2a-49f5-993b-5472252c82c6 to worker[183] qsize 0
2021-09-27 11:53:20,610 DEBUG awx.main.dispatch task f69224f7-1c2a-49f5-993b-5472252c82c6 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-09-27 11:53:20,613 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-09-27 11:53:20,653 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4ff408a1-cf7e-438c-a69f-08d5fffbe39b, queue=dev-awx-0)
[2021-09-27 11:53:20,653: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4ff408a1-cf7e-438c-a69f-08d5fffbe39b, queue=dev-awx-0)
2021-09-27 11:53:20,662 DEBUG awx.main.dispatch task f69224f7-1c2a-49f5-993b-5472252c82c6 is finished
2021-09-27 11:53:20,743 DEBUG awx.main.dispatch delivered 4ff408a1-cf7e-438c-a69f-08d5fffbe39b to worker[183] qsize 0
2021-09-27 11:53:20,746 DEBUG awx.main.dispatch task 4ff408a1-cf7e-438c-a69f-08d5fffbe39b starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:53:20,748 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:53:20,858 DEBUG awx.main.dispatch task 4ff408a1-cf7e-438c-a69f-08d5fffbe39b is finished
2021-09-27 11:53:21,966 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(1022e615-9dea-4ca5-ac82-689c2f550269, queue=dev-awx-0)
[2021-09-27 11:53:21,966: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(1022e615-9dea-4ca5-ac82-689c2f550269, queue=dev-awx-0)
2021-09-27 11:53:21,990 DEBUG awx.main.dispatch delivered 1022e615-9dea-4ca5-ac82-689c2f550269 to worker[180] qsize 0
2021-09-27 11:53:21,994 DEBUG awx.main.dispatch task 1022e615-9dea-4ca5-ac82-689c2f550269 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:53:22,056 DEBUG awx.main.dispatch task 1022e615-9dea-4ca5-ac82-689c2f550269 is finished
2021-09-27 11:53:23,405 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(90fde78a-59ac-40f1-b627-2126a1e90d64, queue=awx_private_queue)
[2021-09-27 11:53:23,405: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(90fde78a-59ac-40f1-b627-2126a1e90d64, queue=awx_private_queue)
2021-09-27 11:53:23,425 DEBUG awx.main.dispatch delivered 90fde78a-59ac-40f1-b627-2126a1e90d64 to worker[181] qsize 0
2021-09-27 11:53:23,426 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(be2de1d7-aadb-429f-a6f3-b171440f6999, queue=awx_private_queue)
[2021-09-27 11:53:23,426: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(be2de1d7-aadb-429f-a6f3-b171440f6999, queue=awx_private_queue)
2021-09-27 11:53:23,428 DEBUG awx.main.dispatch task 90fde78a-59ac-40f1-b627-2126a1e90d64 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:53:23,429 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:53:23,446 DEBUG awx.main.dispatch delivered be2de1d7-aadb-429f-a6f3-b171440f6999 to worker[180] qsize 0
2021-09-27 11:53:23,448 DEBUG awx.main.dispatch task be2de1d7-aadb-429f-a6f3-b171440f6999 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:53:23,634 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:53:23,644 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:53:23,648 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:52:53.872859+00:00
2021-09-27 11:53:23,667 DEBUG awx.main.dispatch task be2de1d7-aadb-429f-a6f3-b171440f6999 is finished
2021-09-27 11:53:23,684 DEBUG awx.main.dispatch task 90fde78a-59ac-40f1-b627-2126a1e90d64 is finished
2021-09-27 11:53:43,406 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(8e03cffb-f454-40f0-a331-ffb7dd6f50ce, queue=awx_private_queue)
[2021-09-27 11:53:43,406: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(8e03cffb-f454-40f0-a331-ffb7dd6f50ce, queue=awx_private_queue)
2021-09-27 11:53:43,434 DEBUG awx.main.dispatch delivered 8e03cffb-f454-40f0-a331-ffb7dd6f50ce to worker[181] qsize 0
2021-09-27 11:53:43,439 DEBUG awx.main.dispatch task 8e03cffb-f454-40f0-a331-ffb7dd6f50ce starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:53:43,441 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:53:43,728 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:53:43,827 DEBUG awx.main.dispatch task 8e03cffb-f454-40f0-a331-ffb7dd6f50ce is finished
2021-09-27 11:53:53,427 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8bf91528-87b3-4233-97b3-83277e9c98a3, queue=awx_private_queue)
[2021-09-27 11:53:53,427: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8bf91528-87b3-4233-97b3-83277e9c98a3, queue=awx_private_queue)
2021-09-27 11:53:53,778 DEBUG awx.main.dispatch delivered 8bf91528-87b3-4233-97b3-83277e9c98a3 to worker[182] qsize 0
2021-09-27 11:53:53,781 DEBUG awx.main.dispatch task 8bf91528-87b3-4233-97b3-83277e9c98a3 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:53:53,852 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:53:53,857 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:53:23.645050+00:00
2021-09-27 11:53:53,963 DEBUG awx.main.dispatch task 8bf91528-87b3-4233-97b3-83277e9c98a3 is finished
RESULT 2
OKREADY
2021-09-27 11:54:03,405 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(e6444622-578f-4ef0-ae4f-330116dda578, queue=awx_private_queue)
[2021-09-27 11:54:03,405: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(e6444622-578f-4ef0-ae4f-330116dda578, queue=awx_private_queue)
2021-09-27 11:54:03,421 DEBUG awx.main.dispatch delivered e6444622-578f-4ef0-ae4f-330116dda578 to worker[183] qsize 0
2021-09-27 11:54:03,424 DEBUG awx.main.dispatch task e6444622-578f-4ef0-ae4f-330116dda578 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:54:03,426 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:54:03,495 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:54:03,555 DEBUG awx.main.dispatch task e6444622-578f-4ef0-ae4f-330116dda578 is finished
2021-09-27 11:54:20,658 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(3b58cccb-7536-4fb5-9a17-a72bb9db498f, queue=dev-awx-0)
[2021-09-27 11:54:20,658: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(3b58cccb-7536-4fb5-9a17-a72bb9db498f, queue=dev-awx-0)
2021-09-27 11:54:20,704 DEBUG awx.main.dispatch delivered 3b58cccb-7536-4fb5-9a17-a72bb9db498f to worker[180] qsize 0
2021-09-27 11:54:20,707 DEBUG awx.main.dispatch task 3b58cccb-7536-4fb5-9a17-a72bb9db498f starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:54:20,709 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:54:20,759 DEBUG awx.main.dispatch task 3b58cccb-7536-4fb5-9a17-a72bb9db498f is finished
2021-09-27 11:54:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(916f7f65-563c-4e77-a62d-c3b6e1aa0085, queue=dev-awx-0)
[2021-09-27 11:54:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(916f7f65-563c-4e77-a62d-c3b6e1aa0085, queue=dev-awx-0)
2021-09-27 11:54:21,987 DEBUG awx.main.dispatch delivered 916f7f65-563c-4e77-a62d-c3b6e1aa0085 to worker[182] qsize 0
2021-09-27 11:54:21,997 DEBUG awx.main.dispatch task 916f7f65-563c-4e77-a62d-c3b6e1aa0085 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:54:22,115 DEBUG awx.main.dispatch task 916f7f65-563c-4e77-a62d-c3b6e1aa0085 is finished
2021-09-27 11:54:23,406 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a6094d8e-6820-491f-ab2b-27afd202989f, queue=awx_private_queue)
[2021-09-27 11:54:23,406: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a6094d8e-6820-491f-ab2b-27afd202989f, queue=awx_private_queue)
2021-09-27 11:54:23,446 DEBUG awx.main.dispatch delivered a6094d8e-6820-491f-ab2b-27afd202989f to worker[182] qsize 0
2021-09-27 11:54:23,447 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0ef4e7fd-07ed-4ee1-9019-70db2a366bc4, queue=awx_private_queue)
[2021-09-27 11:54:23,447: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0ef4e7fd-07ed-4ee1-9019-70db2a366bc4, queue=awx_private_queue)
2021-09-27 11:54:23,449 DEBUG awx.main.dispatch task a6094d8e-6820-491f-ab2b-27afd202989f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:54:23,451 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:54:23,469 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:54:23,476 DEBUG awx.main.dispatch delivered 0ef4e7fd-07ed-4ee1-9019-70db2a366bc4 to worker[183] qsize 0
2021-09-27 11:54:23,479 DEBUG awx.main.dispatch task 0ef4e7fd-07ed-4ee1-9019-70db2a366bc4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:54:23,508 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:54:23,509 DEBUG awx.main.dispatch task a6094d8e-6820-491f-ab2b-27afd202989f is finished
2021-09-27 11:54:23,514 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:53:53.852624+00:00
2021-09-27 11:54:23,531 DEBUG awx.main.dispatch task 0ef4e7fd-07ed-4ee1-9019-70db2a366bc4 is finished
2021-09-27 11:54:43,415 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3190aa5f-2b1e-466f-81f0-6374ee9fe297, queue=awx_private_queue)
[2021-09-27 11:54:43,415: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3190aa5f-2b1e-466f-81f0-6374ee9fe297, queue=awx_private_queue)
2021-09-27 11:54:43,435 DEBUG awx.main.dispatch delivered 3190aa5f-2b1e-466f-81f0-6374ee9fe297 to worker[181] qsize 0
2021-09-27 11:54:43,439 DEBUG awx.main.dispatch task 3190aa5f-2b1e-466f-81f0-6374ee9fe297 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:54:43,443 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:54:53,447 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(facc7a81-7cde-4baa-b3bb-c54b210d5e4c, queue=awx_private_queue)
[2021-09-27 11:54:53,447: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(facc7a81-7cde-4baa-b3bb-c54b210d5e4c, queue=awx_private_queue)
2021-09-27 11:54:53,463 DEBUG awx.main.dispatch delivered facc7a81-7cde-4baa-b3bb-c54b210d5e4c to worker[182] qsize 0
2021-09-27 11:54:53,465 DEBUG awx.main.dispatch task facc7a81-7cde-4baa-b3bb-c54b210d5e4c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:54:56,883 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:54:58,624 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:54:23.508434+00:00
2021-09-27 11:54:58,816 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:54:59,348 DEBUG awx.main.dispatch task 3190aa5f-2b1e-466f-81f0-6374ee9fe297 is finished
2021-09-27 11:54:59,289 DEBUG awx.main.dispatch task facc7a81-7cde-4baa-b3bb-c54b210d5e4c is finished
RESULT 2
OKREADY
2021-09-27 11:55:03,414 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(57fdf133-e88e-4dba-90bc-7e1e842e393b, queue=awx_private_queue)
[2021-09-27 11:55:03,414: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(57fdf133-e88e-4dba-90bc-7e1e842e393b, queue=awx_private_queue)
2021-09-27 11:55:03,430 DEBUG awx.main.dispatch delivered 57fdf133-e88e-4dba-90bc-7e1e842e393b to worker[180] qsize 0
2021-09-27 11:55:03,433 DEBUG awx.main.dispatch task 57fdf133-e88e-4dba-90bc-7e1e842e393b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:55:03,436 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:55:04,469 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:55:04,471 DEBUG awx.main.dispatch task 57fdf133-e88e-4dba-90bc-7e1e842e393b is finished
2021-09-27 11:55:20,662 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4fecc530-ec99-42be-9954-b3f061f8a631, queue=dev-awx-0)
[2021-09-27 11:55:20,662: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4fecc530-ec99-42be-9954-b3f061f8a631, queue=dev-awx-0)
2021-09-27 11:55:21,026 DEBUG awx.main.dispatch delivered 4fecc530-ec99-42be-9954-b3f061f8a631 to worker[182] qsize 0
2021-09-27 11:55:21,030 DEBUG awx.main.dispatch task 4fecc530-ec99-42be-9954-b3f061f8a631 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:55:21,032 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:55:21,966 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(eb5f1b60-8bb1-47fe-8d8c-c37eabb2c6ab, queue=dev-awx-0)
[2021-09-27 11:55:21,966: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(eb5f1b60-8bb1-47fe-8d8c-c37eabb2c6ab, queue=dev-awx-0)
2021-09-27 11:55:21,982 DEBUG awx.main.dispatch delivered eb5f1b60-8bb1-47fe-8d8c-c37eabb2c6ab to worker[181] qsize 0
2021-09-27 11:55:21,984 DEBUG awx.main.dispatch task eb5f1b60-8bb1-47fe-8d8c-c37eabb2c6ab starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:55:22,069 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:55:22,071 DEBUG awx.main.dispatch task 4fecc530-ec99-42be-9954-b3f061f8a631 is finished
2021-09-27 11:55:22,993 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:55:22,998 DEBUG awx.main.dispatch task eb5f1b60-8bb1-47fe-8d8c-c37eabb2c6ab is finished
2021-09-27 11:55:23,414 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(793c6297-102d-486c-858e-67771d4a617f, queue=awx_private_queue)
[2021-09-27 11:55:23,414: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(793c6297-102d-486c-858e-67771d4a617f, queue=awx_private_queue)
2021-09-27 11:55:23,428 DEBUG awx.main.dispatch delivered 793c6297-102d-486c-858e-67771d4a617f to worker[181] qsize 0
2021-09-27 11:55:23,430 DEBUG awx.main.dispatch task 793c6297-102d-486c-858e-67771d4a617f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:55:23,431 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:55:23,447 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0f168cc2-8ffc-4ab1-814a-6044c1195ed7, queue=awx_private_queue)
[2021-09-27 11:55:23,447: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0f168cc2-8ffc-4ab1-814a-6044c1195ed7, queue=awx_private_queue)
2021-09-27 11:55:23,460 DEBUG awx.main.dispatch delivered 0f168cc2-8ffc-4ab1-814a-6044c1195ed7 to worker[180] qsize 0
2021-09-27 11:55:23,462 DEBUG awx.main.dispatch task 0f168cc2-8ffc-4ab1-814a-6044c1195ed7 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:55:24,469 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:55:24,471 DEBUG awx.main.dispatch task 793c6297-102d-486c-858e-67771d4a617f is finished
2021-09-27 11:55:24,497 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:55:24,500 DEBUG awx.main.dispatch task 0f168cc2-8ffc-4ab1-814a-6044c1195ed7 is finished
2021-09-27 11:55:43,422 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d7f00cb3-8f21-4825-84ec-6275c1403ca3, queue=awx_private_queue)
[2021-09-27 11:55:43,422: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d7f00cb3-8f21-4825-84ec-6275c1403ca3, queue=awx_private_queue)
2021-09-27 11:55:43,443 DEBUG awx.main.dispatch delivered d7f00cb3-8f21-4825-84ec-6275c1403ca3 to worker[182] qsize 0
2021-09-27 11:55:43,446 DEBUG awx.main.dispatch task d7f00cb3-8f21-4825-84ec-6275c1403ca3 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:55:43,448 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:55:44,465 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:55:44,467 DEBUG awx.main.dispatch task d7f00cb3-8f21-4825-84ec-6275c1403ca3 is finished
2021-09-27 11:55:53,447 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6352f27b-284a-4b50-8dba-563cb4200cea, queue=awx_private_queue)
[2021-09-27 11:55:53,447: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6352f27b-284a-4b50-8dba-563cb4200cea, queue=awx_private_queue)
2021-09-27 11:55:53,465 DEBUG awx.main.dispatch delivered 6352f27b-284a-4b50-8dba-563cb4200cea to worker[180] qsize 0
2021-09-27 11:55:53,466 DEBUG awx.main.dispatch task 6352f27b-284a-4b50-8dba-563cb4200cea starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:55:53,489 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:55:53,495 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:54:56.883449+00:00
2021-09-27 11:55:53,530 DEBUG awx.main.dispatch task 6352f27b-284a-4b50-8dba-563cb4200cea is finished
RESULT 2
OKREADY
2021-09-27 11:56:03,422 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(95284ccb-a861-43dc-b024-787b8a19a2c5, queue=awx_private_queue)
[2021-09-27 11:56:03,422: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(95284ccb-a861-43dc-b024-787b8a19a2c5, queue=awx_private_queue)
2021-09-27 11:56:03,487 DEBUG awx.main.dispatch delivered 95284ccb-a861-43dc-b024-787b8a19a2c5 to worker[181] qsize 0
2021-09-27 11:56:03,679 DEBUG awx.main.dispatch task 95284ccb-a861-43dc-b024-787b8a19a2c5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:56:03,681 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:56:03,757 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:56:03,874 DEBUG awx.main.dispatch task 95284ccb-a861-43dc-b024-787b8a19a2c5 is finished
2021-09-27 11:56:20,670 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(952b8316-6c71-4c02-82ab-dbc14b6282fe, queue=dev-awx-0)
[2021-09-27 11:56:20,670: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(952b8316-6c71-4c02-82ab-dbc14b6282fe, queue=dev-awx-0)
2021-09-27 11:56:20,758 DEBUG awx.main.dispatch delivered 952b8316-6c71-4c02-82ab-dbc14b6282fe to worker[183] qsize 0
2021-09-27 11:56:20,762 DEBUG awx.main.dispatch task 952b8316-6c71-4c02-82ab-dbc14b6282fe starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:56:20,765 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:56:21,777 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:56:21,784 DEBUG awx.main.dispatch task 952b8316-6c71-4c02-82ab-dbc14b6282fe is finished
2021-09-27 11:56:21,966 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2be85bf6-3ef5-4117-9e65-99934905b344, queue=dev-awx-0)
[2021-09-27 11:56:21,966: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2be85bf6-3ef5-4117-9e65-99934905b344, queue=dev-awx-0)
2021-09-27 11:56:21,981 DEBUG awx.main.dispatch delivered 2be85bf6-3ef5-4117-9e65-99934905b344 to worker[181] qsize 0
2021-09-27 11:56:21,984 DEBUG awx.main.dispatch task 2be85bf6-3ef5-4117-9e65-99934905b344 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:56:22,993 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:56:22,996 DEBUG awx.main.dispatch task 2be85bf6-3ef5-4117-9e65-99934905b344 is finished
2021-09-27 11:56:23,447 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(43fb6c44-59a7-427c-8c68-df87993bc76d, queue=awx_private_queue)
[2021-09-27 11:56:23,447: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(43fb6c44-59a7-427c-8c68-df87993bc76d, queue=awx_private_queue)
2021-09-27 11:56:23,462 DEBUG awx.main.dispatch delivered 43fb6c44-59a7-427c-8c68-df87993bc76d to worker[183] qsize 0
2021-09-27 11:56:23,463 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ea365f9e-6475-437e-9738-a636690a274b, queue=awx_private_queue)
[2021-09-27 11:56:23,463: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ea365f9e-6475-437e-9738-a636690a274b, queue=awx_private_queue)
2021-09-27 11:56:23,464 DEBUG awx.main.dispatch task 43fb6c44-59a7-427c-8c68-df87993bc76d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:56:23,477 DEBUG awx.main.dispatch delivered ea365f9e-6475-437e-9738-a636690a274b to worker[180] qsize 0
2021-09-27 11:56:23,480 DEBUG awx.main.dispatch task ea365f9e-6475-437e-9738-a636690a274b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:56:23,483 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:56:24,497 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:56:24,497 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:56:24,499 DEBUG awx.main.dispatch task 43fb6c44-59a7-427c-8c68-df87993bc76d is finished
2021-09-27 11:56:24,499 DEBUG awx.main.dispatch task ea365f9e-6475-437e-9738-a636690a274b is finished
2021-09-27 11:56:43,474 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7cb4bc5d-b557-4a25-8a6d-ba2ec8416d0e, queue=awx_private_queue)
[2021-09-27 11:56:43,474: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7cb4bc5d-b557-4a25-8a6d-ba2ec8416d0e, queue=awx_private_queue)
2021-09-27 11:56:43,489 DEBUG awx.main.dispatch delivered 7cb4bc5d-b557-4a25-8a6d-ba2ec8416d0e to worker[183] qsize 0
2021-09-27 11:56:43,491 DEBUG awx.main.dispatch task 7cb4bc5d-b557-4a25-8a6d-ba2ec8416d0e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:56:43,493 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:56:44,497 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:56:44,501 DEBUG awx.main.dispatch task 7cb4bc5d-b557-4a25-8a6d-ba2ec8416d0e is finished
2021-09-27 11:56:53,449 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(aa11c41a-6f15-4842-bab0-48ef32fc65dc, queue=awx_private_queue)
[2021-09-27 11:56:53,449: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(aa11c41a-6f15-4842-bab0-48ef32fc65dc, queue=awx_private_queue)
2021-09-27 11:56:53,478 DEBUG awx.main.dispatch delivered aa11c41a-6f15-4842-bab0-48ef32fc65dc to worker[181] qsize 0
2021-09-27 11:56:53,497 DEBUG awx.main.dispatch task aa11c41a-6f15-4842-bab0-48ef32fc65dc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:56:54,514 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:56:54,520 DEBUG awx.main.dispatch task aa11c41a-6f15-4842-bab0-48ef32fc65dc is finished
RESULT 2
OKREADY
2021-09-27 11:57:03,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(126a2d97-24b1-4ea2-a41d-7b8d0357a87f, queue=awx_private_queue)
[2021-09-27 11:57:03,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(126a2d97-24b1-4ea2-a41d-7b8d0357a87f, queue=awx_private_queue)
2021-09-27 11:57:04,525 DEBUG awx.main.dispatch delivered 126a2d97-24b1-4ea2-a41d-7b8d0357a87f to worker[180] qsize 0
2021-09-27 11:57:04,766 DEBUG awx.main.dispatch task 126a2d97-24b1-4ea2-a41d-7b8d0357a87f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:57:04,769 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:57:05,777 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:57:05,779 DEBUG awx.main.dispatch task 126a2d97-24b1-4ea2-a41d-7b8d0357a87f is finished
2021-09-27 11:57:20,674 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(8387f732-c83f-4f5d-9614-748c95b5e6bc, queue=dev-awx-0)
[2021-09-27 11:57:20,674: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(8387f732-c83f-4f5d-9614-748c95b5e6bc, queue=dev-awx-0)
2021-09-27 11:57:20,852 DEBUG awx.main.dispatch delivered 8387f732-c83f-4f5d-9614-748c95b5e6bc to worker[182] qsize 0
2021-09-27 11:57:20,855 DEBUG awx.main.dispatch task 8387f732-c83f-4f5d-9614-748c95b5e6bc starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:57:20,859 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:57:21,966 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ec6c8db4-85be-49cc-bb2a-f0abd9b4203c, queue=dev-awx-0)
[2021-09-27 11:57:21,966: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ec6c8db4-85be-49cc-bb2a-f0abd9b4203c, queue=dev-awx-0)
2021-09-27 11:57:21,988 DEBUG awx.main.dispatch delivered ec6c8db4-85be-49cc-bb2a-f0abd9b4203c to worker[181] qsize 0
2021-09-27 11:57:21,991 DEBUG awx.main.dispatch task ec6c8db4-85be-49cc-bb2a-f0abd9b4203c starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:57:22,522 DEBUG awx.main.dispatch task ec6c8db4-85be-49cc-bb2a-f0abd9b4203c is finished
2021-09-27 11:57:22,526 WARNING awx.main.tasks Rejoining the cluster as instance dev-awx-0.
2021-09-27 11:57:22,526 DEBUG awx.main.dispatch task 8387f732-c83f-4f5d-9614-748c95b5e6bc is finished
2021-09-27 11:57:23,448 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8428eff2-7f17-47c2-88ca-8ea74d9085d6, queue=awx_private_queue)
[2021-09-27 11:57:23,448: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8428eff2-7f17-47c2-88ca-8ea74d9085d6, queue=awx_private_queue)
2021-09-27 11:57:23,473 DEBUG awx.main.dispatch delivered 8428eff2-7f17-47c2-88ca-8ea74d9085d6 to worker[183] qsize 0
2021-09-27 11:57:23,474 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(ab3cc80e-a046-45e2-97ef-ddb7ba4408f2, queue=awx_private_queue)
[2021-09-27 11:57:23,474: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(ab3cc80e-a046-45e2-97ef-ddb7ba4408f2, queue=awx_private_queue)
2021-09-27 11:57:23,476 DEBUG awx.main.dispatch task 8428eff2-7f17-47c2-88ca-8ea74d9085d6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:57:23,492 DEBUG awx.main.dispatch delivered ab3cc80e-a046-45e2-97ef-ddb7ba4408f2 to worker[180] qsize 0
2021-09-27 11:57:23,498 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:57:23,496 DEBUG awx.main.dispatch task ab3cc80e-a046-45e2-97ef-ddb7ba4408f2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:57:23,499 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:57:23,505 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:55:53.489398+00:00
2021-09-27 11:57:23,522 DEBUG awx.main.dispatch task 8428eff2-7f17-47c2-88ca-8ea74d9085d6 is finished
2021-09-27 11:57:23,525 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:57:23,568 DEBUG awx.main.dispatch task ab3cc80e-a046-45e2-97ef-ddb7ba4408f2 is finished
2021-09-27 11:57:43,479 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a81dab78-bc73-486d-a13d-048930f075d1, queue=awx_private_queue)
[2021-09-27 11:57:43,479: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a81dab78-bc73-486d-a13d-048930f075d1, queue=awx_private_queue)
2021-09-27 11:57:43,517 DEBUG awx.main.dispatch delivered a81dab78-bc73-486d-a13d-048930f075d1 to worker[182] qsize 0
2021-09-27 11:57:43,523 DEBUG awx.main.dispatch task a81dab78-bc73-486d-a13d-048930f075d1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:57:43,525 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:57:44,561 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:57:44,564 DEBUG awx.main.dispatch task a81dab78-bc73-486d-a13d-048930f075d1 is finished
2021-09-27 11:57:53,448 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(8dddb8a8-6337-4590-b40a-c5fc618f673b, queue=awx_private_queue)
[2021-09-27 11:57:53,448: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(8dddb8a8-6337-4590-b40a-c5fc618f673b, queue=awx_private_queue)
2021-09-27 11:57:53,466 DEBUG awx.main.dispatch delivered 8dddb8a8-6337-4590-b40a-c5fc618f673b to worker[180] qsize 0
2021-09-27 11:57:53,470 DEBUG awx.main.dispatch task 8dddb8a8-6337-4590-b40a-c5fc618f673b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:57:54,481 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:57:54,485 DEBUG awx.main.dispatch task 8dddb8a8-6337-4590-b40a-c5fc618f673b is finished
RESULT 2
OKREADY
2021-09-27 11:58:03,475 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a2ae2a27-1ae5-4b8c-87a4-d6b00857ed32, queue=awx_private_queue)
[2021-09-27 11:58:03,475: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a2ae2a27-1ae5-4b8c-87a4-d6b00857ed32, queue=awx_private_queue)
2021-09-27 11:58:04,518 DEBUG awx.main.dispatch delivered a2ae2a27-1ae5-4b8c-87a4-d6b00857ed32 to worker[181] qsize 0
2021-09-27 11:58:04,534 DEBUG awx.main.dispatch task a2ae2a27-1ae5-4b8c-87a4-d6b00857ed32 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:58:04,544 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:58:05,553 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 11:58:05,556 DEBUG awx.main.dispatch task a2ae2a27-1ae5-4b8c-87a4-d6b00857ed32 is finished
2021-09-27 11:58:20,675 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5b1ddda5-754a-499b-9e02-24f8a14982f7, queue=dev-awx-0)
[2021-09-27 11:58:20,675: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5b1ddda5-754a-499b-9e02-24f8a14982f7, queue=dev-awx-0)
2021-09-27 11:58:21,168 DEBUG awx.main.dispatch delivered 5b1ddda5-754a-499b-9e02-24f8a14982f7 to worker[180] qsize 0
2021-09-27 11:58:21,400 DEBUG awx.main.dispatch task 5b1ddda5-754a-499b-9e02-24f8a14982f7 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:58:21,402 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:58:21,577 DEBUG awx.main.dispatch task 5b1ddda5-754a-499b-9e02-24f8a14982f7 is finished
2021-09-27 11:58:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ed2e1dcc-f696-46e1-b664-642298616f46, queue=dev-awx-0)
[2021-09-27 11:58:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ed2e1dcc-f696-46e1-b664-642298616f46, queue=dev-awx-0)
2021-09-27 11:58:21,993 DEBUG awx.main.dispatch delivered ed2e1dcc-f696-46e1-b664-642298616f46 to worker[183] qsize 0
2021-09-27 11:58:21,997 DEBUG awx.main.dispatch task ed2e1dcc-f696-46e1-b664-642298616f46 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:58:22,115 DEBUG awx.main.dispatch task ed2e1dcc-f696-46e1-b664-642298616f46 is finished
2021-09-27 11:58:23,447 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(69e0b59c-eb68-49b7-a2bf-56aafc1d6bf4, queue=awx_private_queue)
[2021-09-27 11:58:23,447: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(69e0b59c-eb68-49b7-a2bf-56aafc1d6bf4, queue=awx_private_queue)
2021-09-27 11:58:23,466 DEBUG awx.main.dispatch delivered 69e0b59c-eb68-49b7-a2bf-56aafc1d6bf4 to worker[183] qsize 0
2021-09-27 11:58:23,469 DEBUG awx.main.dispatch task 69e0b59c-eb68-49b7-a2bf-56aafc1d6bf4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:58:23,474 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5965e4d7-1b81-467f-93e3-557e1da399d4, queue=awx_private_queue)
[2021-09-27 11:58:23,474: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5965e4d7-1b81-467f-93e3-557e1da399d4, queue=awx_private_queue)
2021-09-27 11:58:23,490 DEBUG awx.main.dispatch delivered 5965e4d7-1b81-467f-93e3-557e1da399d4 to worker[182] qsize 0
2021-09-27 11:58:23,492 DEBUG awx.main.dispatch task 5965e4d7-1b81-467f-93e3-557e1da399d4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:58:23,494 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:58:25,412 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:58:25,451 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:57:23.498908+00:00
2021-09-27 11:58:25,459 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:58:25,544 DEBUG awx.main.dispatch task 5965e4d7-1b81-467f-93e3-557e1da399d4 is finished
2021-09-27 11:58:26,254 DEBUG awx.main.dispatch task 69e0b59c-eb68-49b7-a2bf-56aafc1d6bf4 is finished
2021-09-27 11:58:43,491 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(645c4a21-5438-4a2c-a0f3-04a33063753a, queue=awx_private_queue)
[2021-09-27 11:58:43,491: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(645c4a21-5438-4a2c-a0f3-04a33063753a, queue=awx_private_queue)
2021-09-27 11:58:43,517 DEBUG awx.main.dispatch delivered 645c4a21-5438-4a2c-a0f3-04a33063753a to worker[181] qsize 0
2021-09-27 11:58:43,521 DEBUG awx.main.dispatch task 645c4a21-5438-4a2c-a0f3-04a33063753a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:58:43,525 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:58:43,554 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:58:43,614 DEBUG awx.main.dispatch task 645c4a21-5438-4a2c-a0f3-04a33063753a is finished
2021-09-27 11:58:53,450 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e1308398-7003-44cc-bc3b-25c23a1d88a9, queue=awx_private_queue)
[2021-09-27 11:58:53,450: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e1308398-7003-44cc-bc3b-25c23a1d88a9, queue=awx_private_queue)
2021-09-27 11:58:53,476 DEBUG awx.main.dispatch delivered e1308398-7003-44cc-bc3b-25c23a1d88a9 to worker[181] qsize 0
2021-09-27 11:58:53,481 DEBUG awx.main.dispatch task e1308398-7003-44cc-bc3b-25c23a1d88a9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:58:53,507 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:58:53,514 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:58:25.412641+00:00
2021-09-27 11:58:53,531 DEBUG awx.main.dispatch task e1308398-7003-44cc-bc3b-25c23a1d88a9 is finished
RESULT 2
OKREADY
2021-09-27 11:59:03,492 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fefd0f56-e500-4dbb-96d9-4166b3a4e83b, queue=awx_private_queue)
[2021-09-27 11:59:03,492: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fefd0f56-e500-4dbb-96d9-4166b3a4e83b, queue=awx_private_queue)
2021-09-27 11:59:03,519 DEBUG awx.main.dispatch delivered fefd0f56-e500-4dbb-96d9-4166b3a4e83b to worker[182] qsize 0
2021-09-27 11:59:03,523 DEBUG awx.main.dispatch task fefd0f56-e500-4dbb-96d9-4166b3a4e83b starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:59:03,525 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:59:03,548 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:59:03,584 DEBUG awx.main.dispatch task fefd0f56-e500-4dbb-96d9-4166b3a4e83b is finished
2021-09-27 11:59:20,682 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6072bca0-4432-4bb1-ae9c-97d23104ef21, queue=dev-awx-0)
[2021-09-27 11:59:20,682: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6072bca0-4432-4bb1-ae9c-97d23104ef21, queue=dev-awx-0)
2021-09-27 11:59:20,742 DEBUG awx.main.dispatch delivered 6072bca0-4432-4bb1-ae9c-97d23104ef21 to worker[180] qsize 0
2021-09-27 11:59:20,745 DEBUG awx.main.dispatch task 6072bca0-4432-4bb1-ae9c-97d23104ef21 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 11:59:20,749 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 11:59:20,782 DEBUG awx.main.dispatch task 6072bca0-4432-4bb1-ae9c-97d23104ef21 is finished
2021-09-27 11:59:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(813e00f4-e245-4bc7-a3b2-b1aa69982734, queue=dev-awx-0)
[2021-09-27 11:59:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(813e00f4-e245-4bc7-a3b2-b1aa69982734, queue=dev-awx-0)
2021-09-27 11:59:22,258 DEBUG awx.main.dispatch delivered 813e00f4-e245-4bc7-a3b2-b1aa69982734 to worker[182] qsize 0
2021-09-27 11:59:22,469 DEBUG awx.main.dispatch task 813e00f4-e245-4bc7-a3b2-b1aa69982734 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 11:59:22,484 DEBUG awx.main.dispatch task 813e00f4-e245-4bc7-a3b2-b1aa69982734 is finished
2021-09-27 11:59:23,448 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(dd89a588-9fbf-4b14-bcbe-4153ef25706a, queue=awx_private_queue)
[2021-09-27 11:59:23,448: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(dd89a588-9fbf-4b14-bcbe-4153ef25706a, queue=awx_private_queue)
2021-09-27 11:59:23,470 DEBUG awx.main.dispatch delivered dd89a588-9fbf-4b14-bcbe-4153ef25706a to worker[181] qsize 0
2021-09-27 11:59:23,472 DEBUG awx.main.dispatch task dd89a588-9fbf-4b14-bcbe-4153ef25706a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:59:23,489 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:59:23,493 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:58:53.508125+00:00
2021-09-27 11:59:23,491 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(13c8d754-e81f-4487-8860-c92395a18d4a, queue=awx_private_queue)
[2021-09-27 11:59:23,491: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(13c8d754-e81f-4487-8860-c92395a18d4a, queue=awx_private_queue)
2021-09-27 11:59:23,512 DEBUG awx.main.dispatch delivered 13c8d754-e81f-4487-8860-c92395a18d4a to worker[183] qsize 0
2021-09-27 11:59:23,515 DEBUG awx.main.dispatch task 13c8d754-e81f-4487-8860-c92395a18d4a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:59:23,519 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:59:23,522 DEBUG awx.main.dispatch task dd89a588-9fbf-4b14-bcbe-4153ef25706a is finished
2021-09-27 11:59:23,544 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:59:23,580 DEBUG awx.main.dispatch task 13c8d754-e81f-4487-8860-c92395a18d4a is finished
2021-09-27 11:59:43,501 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a87cdd54-0106-4615-afdb-b53e5c902051, queue=awx_private_queue)
[2021-09-27 11:59:43,501: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a87cdd54-0106-4615-afdb-b53e5c902051, queue=awx_private_queue)
2021-09-27 11:59:43,526 DEBUG awx.main.dispatch delivered a87cdd54-0106-4615-afdb-b53e5c902051 to worker[181] qsize 0
2021-09-27 11:59:43,529 DEBUG awx.main.dispatch task a87cdd54-0106-4615-afdb-b53e5c902051 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 11:59:43,533 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 11:59:43,563 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 11:59:43,810 DEBUG awx.main.dispatch task a87cdd54-0106-4615-afdb-b53e5c902051 is finished
2021-09-27 11:59:53,449 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e366bdb9-e84d-49de-89ca-1d64745dd973, queue=awx_private_queue)
[2021-09-27 11:59:53,449: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e366bdb9-e84d-49de-89ca-1d64745dd973, queue=awx_private_queue)
2021-09-27 11:59:53,474 DEBUG awx.main.dispatch delivered e366bdb9-e84d-49de-89ca-1d64745dd973 to worker[182] qsize 0
2021-09-27 11:59:53,477 DEBUG awx.main.dispatch task e366bdb9-e84d-49de-89ca-1d64745dd973 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 11:59:54,805 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 11:59:54,831 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:59:23.489975+00:00
2021-09-27 11:59:54,944 DEBUG awx.main.dispatch task e366bdb9-e84d-49de-89ca-1d64745dd973 is finished
2021-09-27 12:00:00,002 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(c2573e59-1735-4e5b-879d-59b48d2cf923, queue=awx_private_queue)
[2021-09-27 12:00:00,002: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(c2573e59-1735-4e5b-879d-59b48d2cf923, queue=awx_private_queue)
2021-09-27 12:00:00,038 DEBUG awx.main.dispatch delivered c2573e59-1735-4e5b-879d-59b48d2cf923 to worker[180] qsize 0
RESULT 2
OKREADY
2021-09-27 12:00:00,041 DEBUG awx.main.dispatch task c2573e59-1735-4e5b-879d-59b48d2cf923 starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:00:00,045 DEBUG awx.main.dispatch task c2573e59-1735-4e5b-879d-59b48d2cf923 is finished
2021-09-27 12:00:03,501 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(bd787882-0a22-4158-a22c-660e44288f63, queue=awx_private_queue)
[2021-09-27 12:00:03,501: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(bd787882-0a22-4158-a22c-660e44288f63, queue=awx_private_queue)
2021-09-27 12:00:03,523 DEBUG awx.main.dispatch delivered bd787882-0a22-4158-a22c-660e44288f63 to worker[183] qsize 0
2021-09-27 12:00:03,527 DEBUG awx.main.dispatch task bd787882-0a22-4158-a22c-660e44288f63 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:00:03,531 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:00:10,617 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:00:14,627 DEBUG awx.main.dispatch task bd787882-0a22-4158-a22c-660e44288f63 is finished
2021-09-27 12:00:20,694 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(1b778177-5aa9-4edc-bc18-e47c75c13c4e, queue=dev-awx-0)
[2021-09-27 12:00:20,694: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(1b778177-5aa9-4edc-bc18-e47c75c13c4e, queue=dev-awx-0)
2021-09-27 12:00:20,779 DEBUG awx.main.dispatch delivered 1b778177-5aa9-4edc-bc18-e47c75c13c4e to worker[180] qsize 0
2021-09-27 12:00:20,783 DEBUG awx.main.dispatch task 1b778177-5aa9-4edc-bc18-e47c75c13c4e starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 12:00:20,788 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 12:00:20,846 DEBUG awx.main.dispatch task 1b778177-5aa9-4edc-bc18-e47c75c13c4e is finished
2021-09-27 12:00:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4998a3b5-13a0-4623-a251-759690ff5ec4, queue=dev-awx-0)
[2021-09-27 12:00:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4998a3b5-13a0-4623-a251-759690ff5ec4, queue=dev-awx-0)
2021-09-27 12:00:21,994 DEBUG awx.main.dispatch delivered 4998a3b5-13a0-4623-a251-759690ff5ec4 to worker[182] qsize 0
2021-09-27 12:00:21,997 DEBUG awx.main.dispatch task 4998a3b5-13a0-4623-a251-759690ff5ec4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:00:22,033 DEBUG awx.main.dispatch task 4998a3b5-13a0-4623-a251-759690ff5ec4 is finished
2021-09-27 12:00:23,449 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(50b10d46-5e63-4e02-8504-b53960f164eb, queue=awx_private_queue)
[2021-09-27 12:00:23,449: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(50b10d46-5e63-4e02-8504-b53960f164eb, queue=awx_private_queue)
2021-09-27 12:00:26,542 DEBUG awx.main.dispatch delivered 50b10d46-5e63-4e02-8504-b53960f164eb to worker[183] qsize 0
2021-09-27 12:00:26,737 DEBUG awx.main.dispatch task 50b10d46-5e63-4e02-8504-b53960f164eb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:00:28,312 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 12:00:28,326 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 11:59:54.807137+00:00
2021-09-27 12:00:28,356 DEBUG awx.main.dispatch task 50b10d46-5e63-4e02-8504-b53960f164eb is finished
2021-09-27 12:00:26,544 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(94534b35-37c0-48b9-9676-6ff57d9dcb55, queue=awx_private_queue)
[2021-09-27 12:00:26,544: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(94534b35-37c0-48b9-9676-6ff57d9dcb55, queue=awx_private_queue)
2021-09-27 12:00:28,570 DEBUG awx.main.dispatch delivered 94534b35-37c0-48b9-9676-6ff57d9dcb55 to worker[182] qsize 0
2021-09-27 12:00:28,572 DEBUG awx.main.dispatch task 94534b35-37c0-48b9-9676-6ff57d9dcb55 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:00:28,574 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:00:28,663 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:00:28,942 DEBUG awx.main.dispatch task 94534b35-37c0-48b9-9676-6ff57d9dcb55 is finished
2021-09-27 12:00:46,553 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3e2b2a2c-d96b-45de-91b8-85dba4f99587, queue=awx_private_queue)
[2021-09-27 12:00:46,553: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3e2b2a2c-d96b-45de-91b8-85dba4f99587, queue=awx_private_queue)
2021-09-27 12:00:46,592 DEBUG awx.main.dispatch delivered 3e2b2a2c-d96b-45de-91b8-85dba4f99587 to worker[183] qsize 0
2021-09-27 12:00:46,595 DEBUG awx.main.dispatch task 3e2b2a2c-d96b-45de-91b8-85dba4f99587 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:00:46,597 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:00:53,450 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(cc2a2b5f-2f67-426c-82cf-13d2274911b6, queue=awx_private_queue)
[2021-09-27 12:00:53,450: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(cc2a2b5f-2f67-426c-82cf-13d2274911b6, queue=awx_private_queue)
2021-09-27 12:00:53,474 DEBUG awx.main.dispatch delivered cc2a2b5f-2f67-426c-82cf-13d2274911b6 to worker[180] qsize 0
2021-09-27 12:00:53,478 DEBUG awx.main.dispatch task cc2a2b5f-2f67-426c-82cf-13d2274911b6 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:01:00,002 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(f1eb0967-5d49-45da-924a-7e0a8180524b, queue=awx_private_queue)
[2021-09-27 12:01:00,002: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(f1eb0967-5d49-45da-924a-7e0a8180524b, queue=awx_private_queue)
2021-09-27 12:01:00,027 DEBUG awx.main.dispatch delivered f1eb0967-5d49-45da-924a-7e0a8180524b to worker[181] qsize 0
RESULT 2
OKREADY
2021-09-27 12:01:00,036 DEBUG awx.main.dispatch task f1eb0967-5d49-45da-924a-7e0a8180524b starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:01:00,043 DEBUG awx.main.dispatch task f1eb0967-5d49-45da-924a-7e0a8180524b is finished
2021-09-27 12:01:05,135 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 12:01:05,145 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 12:00:28.313192+00:00
2021-09-27 12:01:05,184 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:01:05,213 DEBUG awx.main.dispatch task cc2a2b5f-2f67-426c-82cf-13d2274911b6 is finished
2021-09-27 12:01:05,258 DEBUG awx.main.dispatch task 3e2b2a2c-d96b-45de-91b8-85dba4f99587 is finished
2021-09-27 12:01:06,552 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(584253f7-22de-4136-a084-7cb70f7e59c4, queue=awx_private_queue)
[2021-09-27 12:01:06,552: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(584253f7-22de-4136-a084-7cb70f7e59c4, queue=awx_private_queue)
2021-09-27 12:01:06,575 DEBUG awx.main.dispatch delivered 584253f7-22de-4136-a084-7cb70f7e59c4 to worker[181] qsize 0
2021-09-27 12:01:06,578 DEBUG awx.main.dispatch task 584253f7-22de-4136-a084-7cb70f7e59c4 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:01:06,580 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:01:07,605 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:01:07,608 DEBUG awx.main.dispatch task 584253f7-22de-4136-a084-7cb70f7e59c4 is finished
2021-09-27 12:01:20,691 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(04054acd-a197-4ef8-b4b4-f770e1e2d25a, queue=dev-awx-0)
[2021-09-27 12:01:20,691: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(04054acd-a197-4ef8-b4b4-f770e1e2d25a, queue=dev-awx-0)
2021-09-27 12:01:20,885 DEBUG awx.main.dispatch delivered 04054acd-a197-4ef8-b4b4-f770e1e2d25a to worker[180] qsize 0
2021-09-27 12:01:20,888 DEBUG awx.main.dispatch task 04054acd-a197-4ef8-b4b4-f770e1e2d25a starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 12:01:20,891 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 12:01:21,905 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:01:21,909 DEBUG awx.main.dispatch task 04054acd-a197-4ef8-b4b4-f770e1e2d25a is finished
2021-09-27 12:01:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(a945ebdd-4fc3-4818-9ab9-c9ac784c3ec7, queue=dev-awx-0)
[2021-09-27 12:01:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(a945ebdd-4fc3-4818-9ab9-c9ac784c3ec7, queue=dev-awx-0)
2021-09-27 12:01:21,991 DEBUG awx.main.dispatch delivered a945ebdd-4fc3-4818-9ab9-c9ac784c3ec7 to worker[183] qsize 0
2021-09-27 12:01:21,994 DEBUG awx.main.dispatch task a945ebdd-4fc3-4818-9ab9-c9ac784c3ec7 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:01:23,025 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:01:23,028 DEBUG awx.main.dispatch task a945ebdd-4fc3-4818-9ab9-c9ac784c3ec7 is finished
2021-09-27 12:01:23,450 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(26544950-866a-43ba-825a-655279335a02, queue=awx_private_queue)
[2021-09-27 12:01:23,450: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(26544950-866a-43ba-825a-655279335a02, queue=awx_private_queue)
2021-09-27 12:01:23,472 DEBUG awx.main.dispatch delivered 26544950-866a-43ba-825a-655279335a02 to worker[181] qsize 0
2021-09-27 12:01:23,475 DEBUG awx.main.dispatch task 26544950-866a-43ba-825a-655279335a02 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:01:24,497 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:01:24,501 DEBUG awx.main.dispatch task 26544950-866a-43ba-825a-655279335a02 is finished
2021-09-27 12:01:26,553 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(c4e2e2f4-82ad-4e39-839b-5efeb5e9a3e7, queue=awx_private_queue)
[2021-09-27 12:01:26,553: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(c4e2e2f4-82ad-4e39-839b-5efeb5e9a3e7, queue=awx_private_queue)
2021-09-27 12:01:26,575 DEBUG awx.main.dispatch delivered c4e2e2f4-82ad-4e39-839b-5efeb5e9a3e7 to worker[183] qsize 0
2021-09-27 12:01:26,580 DEBUG awx.main.dispatch task c4e2e2f4-82ad-4e39-839b-5efeb5e9a3e7 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:01:26,583 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:01:27,601 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:01:27,604 DEBUG awx.main.dispatch task c4e2e2f4-82ad-4e39-839b-5efeb5e9a3e7 is finished
2021-09-27 12:01:46,572 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4defc6ab-eacd-444e-a21c-730529de3757, queue=awx_private_queue)
[2021-09-27 12:01:46,572: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4defc6ab-eacd-444e-a21c-730529de3757, queue=awx_private_queue)
2021-09-27 12:01:47,632 DEBUG awx.main.dispatch delivered 4defc6ab-eacd-444e-a21c-730529de3757 to worker[180] qsize 0
2021-09-27 12:01:47,964 DEBUG awx.main.dispatch task 4defc6ab-eacd-444e-a21c-730529de3757 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:01:47,967 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:01:48,977 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:01:48,980 DEBUG awx.main.dispatch task 4defc6ab-eacd-444e-a21c-730529de3757 is finished
2021-09-27 12:01:53,451 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(4300b4b9-5c8a-4bc9-8c87-57984f867fdc, queue=awx_private_queue)
[2021-09-27 12:01:53,451: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(4300b4b9-5c8a-4bc9-8c87-57984f867fdc, queue=awx_private_queue)
2021-09-27 12:01:53,477 DEBUG awx.main.dispatch delivered 4300b4b9-5c8a-4bc9-8c87-57984f867fdc to worker[180] qsize 0
2021-09-27 12:01:53,480 DEBUG awx.main.dispatch task 4300b4b9-5c8a-4bc9-8c87-57984f867fdc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:01:54,513 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:01:54,517 DEBUG awx.main.dispatch task 4300b4b9-5c8a-4bc9-8c87-57984f867fdc is finished
2021-09-27 12:02:00,002 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(44e304e9-ecd6-4ce6-b8bc-572d29ae1776, queue=awx_private_queue)
[2021-09-27 12:02:00,002: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(44e304e9-ecd6-4ce6-b8bc-572d29ae1776, queue=awx_private_queue)
2021-09-27 12:02:00,027 DEBUG awx.main.dispatch delivered 44e304e9-ecd6-4ce6-b8bc-572d29ae1776 to worker[182] qsize 0
RESULT 2
OKREADY
2021-09-27 12:02:00,031 DEBUG awx.main.dispatch task 44e304e9-ecd6-4ce6-b8bc-572d29ae1776 starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:02:02,098 DEBUG awx.main.dispatch task 44e304e9-ecd6-4ce6-b8bc-572d29ae1776 is finished
2021-09-27 12:02:06,570 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(1391adea-f7c3-4a97-9b62-6de1cb2d0e9e, queue=awx_private_queue)
[2021-09-27 12:02:06,570: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(1391adea-f7c3-4a97-9b62-6de1cb2d0e9e, queue=awx_private_queue)
2021-09-27 12:02:06,595 DEBUG awx.main.dispatch delivered 1391adea-f7c3-4a97-9b62-6de1cb2d0e9e to worker[183] qsize 0
2021-09-27 12:02:06,599 DEBUG awx.main.dispatch task 1391adea-f7c3-4a97-9b62-6de1cb2d0e9e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:02:07,640 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:02:08,657 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:02:08,661 DEBUG awx.main.dispatch task 1391adea-f7c3-4a97-9b62-6de1cb2d0e9e is finished
2021-09-27 12:02:20,692 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(9ef68dc2-c2ac-436f-9aa8-377034ffbfcd, queue=dev-awx-0)
[2021-09-27 12:02:20,692: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(9ef68dc2-c2ac-436f-9aa8-377034ffbfcd, queue=dev-awx-0)
2021-09-27 12:02:20,967 DEBUG awx.main.dispatch delivered 9ef68dc2-c2ac-436f-9aa8-377034ffbfcd to worker[181] qsize 0
2021-09-27 12:02:21,968 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(91affeaa-4468-4c7c-9d58-0f3312d6e371, queue=dev-awx-0)
[2021-09-27 12:02:21,968: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(91affeaa-4468-4c7c-9d58-0f3312d6e371, queue=dev-awx-0)
2021-09-27 12:02:21,993 DEBUG awx.main.dispatch delivered 91affeaa-4468-4c7c-9d58-0f3312d6e371 to worker[182] qsize 0
2021-09-27 12:02:21,996 DEBUG awx.main.dispatch task 91affeaa-4468-4c7c-9d58-0f3312d6e371 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:02:20,971 DEBUG awx.main.dispatch task 9ef68dc2-c2ac-436f-9aa8-377034ffbfcd starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 12:02:22,008 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 12:02:23,025 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:02:23,025 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:02:23,028 DEBUG awx.main.dispatch task 9ef68dc2-c2ac-436f-9aa8-377034ffbfcd is finished
2021-09-27 12:02:23,028 DEBUG awx.main.dispatch task 91affeaa-4468-4c7c-9d58-0f3312d6e371 is finished
2021-09-27 12:02:23,451 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(fe02ebe3-ab6f-4d9b-bfae-e5d1e5c8ac24, queue=awx_private_queue)
[2021-09-27 12:02:23,451: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(fe02ebe3-ab6f-4d9b-bfae-e5d1e5c8ac24, queue=awx_private_queue)
2021-09-27 12:02:23,475 DEBUG awx.main.dispatch delivered fe02ebe3-ab6f-4d9b-bfae-e5d1e5c8ac24 to worker[182] qsize 0
2021-09-27 12:02:23,478 DEBUG awx.main.dispatch task fe02ebe3-ab6f-4d9b-bfae-e5d1e5c8ac24 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:02:24,497 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:02:24,501 DEBUG awx.main.dispatch task fe02ebe3-ab6f-4d9b-bfae-e5d1e5c8ac24 is finished
2021-09-27 12:02:26,570 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a034b8c0-b11b-4ecc-93c2-cc273af39f63, queue=awx_private_queue)
[2021-09-27 12:02:26,570: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a034b8c0-b11b-4ecc-93c2-cc273af39f63, queue=awx_private_queue)
2021-09-27 12:02:27,627 DEBUG awx.main.dispatch delivered a034b8c0-b11b-4ecc-93c2-cc273af39f63 to worker[180] qsize 0
2021-09-27 12:02:27,937 DEBUG awx.main.dispatch task a034b8c0-b11b-4ecc-93c2-cc273af39f63 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:02:27,940 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:02:28,945 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:02:28,948 DEBUG awx.main.dispatch task a034b8c0-b11b-4ecc-93c2-cc273af39f63 is finished
2021-09-27 12:02:46,575 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0d868163-2fb0-45e0-919c-290e130d58da, queue=awx_private_queue)
[2021-09-27 12:02:46,575: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0d868163-2fb0-45e0-919c-290e130d58da, queue=awx_private_queue)
2021-09-27 12:02:46,600 DEBUG awx.main.dispatch delivered 0d868163-2fb0-45e0-919c-290e130d58da to worker[182] qsize 0
2021-09-27 12:02:46,603 DEBUG awx.main.dispatch task 0d868163-2fb0-45e0-919c-290e130d58da starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:02:46,606 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:02:46,659 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:02:46,744 DEBUG awx.main.dispatch task 0d868163-2fb0-45e0-919c-290e130d58da is finished
2021-09-27 12:02:53,451 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d67c69a8-5fd3-42b7-a40c-4073844c6d8b, queue=awx_private_queue)
[2021-09-27 12:02:53,451: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d67c69a8-5fd3-42b7-a40c-4073844c6d8b, queue=awx_private_queue)
2021-09-27 12:02:53,492 DEBUG awx.main.dispatch delivered d67c69a8-5fd3-42b7-a40c-4073844c6d8b to worker[181] qsize 0
2021-09-27 12:02:53,495 DEBUG awx.main.dispatch task d67c69a8-5fd3-42b7-a40c-4073844c6d8b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:02:53,583 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 12:02:53,599 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 12:01:05.135308+00:00
2021-09-27 12:02:53,686 DEBUG awx.main.dispatch task d67c69a8-5fd3-42b7-a40c-4073844c6d8b is finished
2021-09-27 12:03:00,001 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(4f707dc7-0ad5-42b2-930a-a5fb843afe8a, queue=awx_private_queue)
[2021-09-27 12:03:00,001: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(4f707dc7-0ad5-42b2-930a-a5fb843afe8a, queue=awx_private_queue)
2021-09-27 12:03:00,026 DEBUG awx.main.dispatch delivered 4f707dc7-0ad5-42b2-930a-a5fb843afe8a to worker[180] qsize 0
RESULT 2
OKREADY
2021-09-27 12:03:00,029 DEBUG awx.main.dispatch task 4f707dc7-0ad5-42b2-930a-a5fb843afe8a starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:03:00,034 DEBUG awx.main.dispatch task 4f707dc7-0ad5-42b2-930a-a5fb843afe8a is finished
2021-09-27 12:03:06,579 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cbdb7d0f-864d-41f1-8f9a-e8db51903f78, queue=awx_private_queue)
[2021-09-27 12:03:06,579: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cbdb7d0f-864d-41f1-8f9a-e8db51903f78, queue=awx_private_queue)
2021-09-27 12:03:06,610 DEBUG awx.main.dispatch delivered cbdb7d0f-864d-41f1-8f9a-e8db51903f78 to worker[182] qsize 0
2021-09-27 12:03:06,616 DEBUG awx.main.dispatch task cbdb7d0f-864d-41f1-8f9a-e8db51903f78 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:03:06,621 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:03:06,646 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:03:06,682 DEBUG awx.main.dispatch task cbdb7d0f-864d-41f1-8f9a-e8db51903f78 is finished
2021-09-27 12:03:20,588 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(43522949-4b77-429e-abcf-85b96e1dc465, queue=dev-awx-0)
[2021-09-27 12:03:20,588: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(43522949-4b77-429e-abcf-85b96e1dc465, queue=dev-awx-0)
2021-09-27 12:03:20,611 DEBUG awx.main.dispatch delivered 43522949-4b77-429e-abcf-85b96e1dc465 to worker[183] qsize 0
2021-09-27 12:03:20,615 DEBUG awx.main.dispatch task 43522949-4b77-429e-abcf-85b96e1dc465 starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-09-27 12:03:20,621 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-09-27 12:03:20,646 DEBUG awx.main.dispatch task 43522949-4b77-429e-abcf-85b96e1dc465 is finished
2021-09-27 12:03:20,691 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(4d9ff806-fef9-4a1d-9fee-68475ba8fa98, queue=dev-awx-0)
[2021-09-27 12:03:20,691: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(4d9ff806-fef9-4a1d-9fee-68475ba8fa98, queue=dev-awx-0)
2021-09-27 12:03:20,732 DEBUG awx.main.dispatch delivered 4d9ff806-fef9-4a1d-9fee-68475ba8fa98 to worker[182] qsize 0
2021-09-27 12:03:20,735 DEBUG awx.main.dispatch task 4d9ff806-fef9-4a1d-9fee-68475ba8fa98 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 12:03:20,737 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 12:03:20,775 WARNING awx.main.tasks Rejoining the cluster as instance dev-awx-0.
2021-09-27 12:03:20,775 DEBUG awx.main.dispatch task 4d9ff806-fef9-4a1d-9fee-68475ba8fa98 is finished
2021-09-27 12:03:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(d5399ab5-7bbb-4334-8d5b-669fc0a71ad4, queue=dev-awx-0)
[2021-09-27 12:03:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(d5399ab5-7bbb-4334-8d5b-669fc0a71ad4, queue=dev-awx-0)
2021-09-27 12:03:21,986 DEBUG awx.main.dispatch delivered d5399ab5-7bbb-4334-8d5b-669fc0a71ad4 to worker[183] qsize 0
2021-09-27 12:03:21,989 DEBUG awx.main.dispatch task d5399ab5-7bbb-4334-8d5b-669fc0a71ad4 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:03:22,017 DEBUG awx.main.dispatch task d5399ab5-7bbb-4334-8d5b-669fc0a71ad4 is finished
2021-09-27 12:03:23,451 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2b9f7a9c-1844-4821-83e4-f6edd549502d, queue=awx_private_queue)
[2021-09-27 12:03:23,451: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2b9f7a9c-1844-4821-83e4-f6edd549502d, queue=awx_private_queue)
2021-09-27 12:03:23,475 DEBUG awx.main.dispatch delivered 2b9f7a9c-1844-4821-83e4-f6edd549502d to worker[183] qsize 0
2021-09-27 12:03:23,478 DEBUG awx.main.dispatch task 2b9f7a9c-1844-4821-83e4-f6edd549502d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:03:24,623 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 12:03:24,632 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 12:02:53.583382+00:00
2021-09-27 12:03:24,668 DEBUG awx.main.dispatch task 2b9f7a9c-1844-4821-83e4-f6edd549502d is finished
2021-09-27 12:03:26,575 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(045a8850-9b8c-473a-a36d-478a898c9610, queue=awx_private_queue)
[2021-09-27 12:03:26,575: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(045a8850-9b8c-473a-a36d-478a898c9610, queue=awx_private_queue)
2021-09-27 12:03:26,601 DEBUG awx.main.dispatch delivered 045a8850-9b8c-473a-a36d-478a898c9610 to worker[183] qsize 0
2021-09-27 12:03:26,604 DEBUG awx.main.dispatch task 045a8850-9b8c-473a-a36d-478a898c9610 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:03:26,605 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:03:28,022 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:03:28,118 DEBUG awx.main.dispatch task 045a8850-9b8c-473a-a36d-478a898c9610 is finished
2021-09-27 12:03:46,582 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7940fb74-4b94-43e5-9e70-4fd80d950bad, queue=awx_private_queue)
[2021-09-27 12:03:46,582: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7940fb74-4b94-43e5-9e70-4fd80d950bad, queue=awx_private_queue)
2021-09-27 12:03:46,608 DEBUG awx.main.dispatch delivered 7940fb74-4b94-43e5-9e70-4fd80d950bad to worker[180] qsize 0
2021-09-27 12:03:50,140 DEBUG awx.main.dispatch task 7940fb74-4b94-43e5-9e70-4fd80d950bad starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:03:50,145 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:03:53,451 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(0c6c03c7-32c1-4b7d-a206-8facd6d9eefb, queue=awx_private_queue)
[2021-09-27 12:03:53,451: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(0c6c03c7-32c1-4b7d-a206-8facd6d9eefb, queue=awx_private_queue)
2021-09-27 12:03:53,474 DEBUG awx.main.dispatch delivered 0c6c03c7-32c1-4b7d-a206-8facd6d9eefb to worker[182] qsize 0
2021-09-27 12:03:53,477 DEBUG awx.main.dispatch task 0c6c03c7-32c1-4b7d-a206-8facd6d9eefb starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:03:55,517 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:03:59,551 DEBUG awx.main.dispatch task 7940fb74-4b94-43e5-9e70-4fd80d950bad is finished
2021-09-27 12:04:00,002 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(406a8d1b-37dd-4249-b5a0-bc02a4909a6c, queue=awx_private_queue)
[2021-09-27 12:04:00,002: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(406a8d1b-37dd-4249-b5a0-bc02a4909a6c, queue=awx_private_queue)
2021-09-27 12:04:00,025 DEBUG awx.main.dispatch delivered 406a8d1b-37dd-4249-b5a0-bc02a4909a6c to worker[180] qsize 0
RESULT 2
OKREADY
2021-09-27 12:04:00,031 DEBUG awx.main.dispatch task 406a8d1b-37dd-4249-b5a0-bc02a4909a6c starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:04:00,035 DEBUG awx.main.dispatch task 406a8d1b-37dd-4249-b5a0-bc02a4909a6c is finished
2021-09-27 12:04:00,354 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 12:04:00,359 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 12:03:24.623585+00:00
2021-09-27 12:04:00,402 DEBUG awx.main.dispatch task 0c6c03c7-32c1-4b7d-a206-8facd6d9eefb is finished
2021-09-27 12:04:06,582 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(4879658d-2a71-43c0-a2e1-9f3be101949d, queue=awx_private_queue)
[2021-09-27 12:04:06,582: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(4879658d-2a71-43c0-a2e1-9f3be101949d, queue=awx_private_queue)
2021-09-27 12:04:06,606 DEBUG awx.main.dispatch delivered 4879658d-2a71-43c0-a2e1-9f3be101949d to worker[181] qsize 0
2021-09-27 12:04:06,610 DEBUG awx.main.dispatch task 4879658d-2a71-43c0-a2e1-9f3be101949d starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:04:06,615 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:04:12,846 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:04:20,694 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6f1c4a6d-0e42-441a-8ddb-bb592dfe5ded, queue=dev-awx-0)
[2021-09-27 12:04:20,694: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6f1c4a6d-0e42-441a-8ddb-bb592dfe5ded, queue=dev-awx-0)
2021-09-27 12:04:21,028 DEBUG awx.main.dispatch delivered 6f1c4a6d-0e42-441a-8ddb-bb592dfe5ded to worker[182] qsize 0
2021-09-27 12:04:21,031 DEBUG awx.main.dispatch task 6f1c4a6d-0e42-441a-8ddb-bb592dfe5ded starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 12:04:21,036 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 12:04:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(c08c916a-89ab-4323-89f0-15fe8bba395b, queue=dev-awx-0)
[2021-09-27 12:04:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(c08c916a-89ab-4323-89f0-15fe8bba395b, queue=dev-awx-0)
2021-09-27 12:04:21,991 DEBUG awx.main.dispatch delivered c08c916a-89ab-4323-89f0-15fe8bba395b to worker[180] qsize 0
2021-09-27 12:04:21,996 DEBUG awx.main.dispatch task c08c916a-89ab-4323-89f0-15fe8bba395b starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:04:22,557 DEBUG awx.main.dispatch task c08c916a-89ab-4323-89f0-15fe8bba395b is finished
2021-09-27 12:04:22,571 DEBUG awx.main.dispatch task 6f1c4a6d-0e42-441a-8ddb-bb592dfe5ded is finished
2021-09-27 12:04:23,452 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(45ae371c-dc58-454c-a51e-eeb084914b7d, queue=awx_private_queue)
[2021-09-27 12:04:23,452: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(45ae371c-dc58-454c-a51e-eeb084914b7d, queue=awx_private_queue)
2021-09-27 12:04:23,477 DEBUG awx.main.dispatch delivered 45ae371c-dc58-454c-a51e-eeb084914b7d to worker[182] qsize 0
2021-09-27 12:04:23,480 DEBUG awx.main.dispatch task 45ae371c-dc58-454c-a51e-eeb084914b7d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:04:23,837 DEBUG awx.main.dispatch task 4879658d-2a71-43c0-a2e1-9f3be101949d is finished
2021-09-27 12:04:24,055 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 12:04:24,058 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 12:04:00.355122+00:00
2021-09-27 12:04:24,100 DEBUG awx.main.dispatch task 45ae371c-dc58-454c-a51e-eeb084914b7d is finished
2021-09-27 12:04:26,584 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(908d09f3-f124-4f49-bab6-65b8cc89adc6, queue=awx_private_queue)
[2021-09-27 12:04:26,584: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(908d09f3-f124-4f49-bab6-65b8cc89adc6, queue=awx_private_queue)
2021-09-27 12:04:26,613 DEBUG awx.main.dispatch delivered 908d09f3-f124-4f49-bab6-65b8cc89adc6 to worker[181] qsize 0
2021-09-27 12:04:26,617 DEBUG awx.main.dispatch task 908d09f3-f124-4f49-bab6-65b8cc89adc6 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:04:26,619 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:04:27,633 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:04:27,637 DEBUG awx.main.dispatch task 908d09f3-f124-4f49-bab6-65b8cc89adc6 is finished
2021-09-27 12:04:46,596 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(19a2a653-f702-4cc0-aa57-04b91592664f, queue=awx_private_queue)
[2021-09-27 12:04:46,596: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(19a2a653-f702-4cc0-aa57-04b91592664f, queue=awx_private_queue)
2021-09-27 12:04:46,625 DEBUG awx.main.dispatch delivered 19a2a653-f702-4cc0-aa57-04b91592664f to worker[180] qsize 0
2021-09-27 12:04:46,631 DEBUG awx.main.dispatch task 19a2a653-f702-4cc0-aa57-04b91592664f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:04:46,634 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:04:47,669 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:04:47,673 DEBUG awx.main.dispatch task 19a2a653-f702-4cc0-aa57-04b91592664f is finished
2021-09-27 12:04:53,451 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f53a36f3-8deb-4fcc-9bc0-81ab5b464dc0, queue=awx_private_queue)
[2021-09-27 12:04:53,451: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f53a36f3-8deb-4fcc-9bc0-81ab5b464dc0, queue=awx_private_queue)
2021-09-27 12:04:54,505 DEBUG awx.main.dispatch delivered f53a36f3-8deb-4fcc-9bc0-81ab5b464dc0 to worker[180] qsize 0
2021-09-27 12:04:55,516 DEBUG awx.main.dispatch task f53a36f3-8deb-4fcc-9bc0-81ab5b464dc0 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:04:57,553 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:04:57,556 DEBUG awx.main.dispatch task f53a36f3-8deb-4fcc-9bc0-81ab5b464dc0 is finished
2021-09-27 12:05:00,002 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(e9e6dfaa-0e12-45c1-bd46-266931b4c2f5, queue=awx_private_queue)
[2021-09-27 12:05:00,002: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(e9e6dfaa-0e12-45c1-bd46-266931b4c2f5, queue=awx_private_queue)
2021-09-27 12:05:00,026 DEBUG awx.main.dispatch delivered e9e6dfaa-0e12-45c1-bd46-266931b4c2f5 to worker[180] qsize 0
2021-09-27 12:05:00,029 DEBUG awx.main.dispatch task e9e6dfaa-0e12-45c1-bd46-266931b4c2f5 starting awx.main.tasks.gather_analytics(*[])
RESULT 2
OKREADY
2021-09-27 12:05:01,042 DEBUG awx.main.dispatch task e9e6dfaa-0e12-45c1-bd46-266931b4c2f5 is finished
2021-09-27 12:05:06,595 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f491b4d1-e451-4240-8e7c-c07f49559c0f, queue=awx_private_queue)
[2021-09-27 12:05:06,595: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f491b4d1-e451-4240-8e7c-c07f49559c0f, queue=awx_private_queue)
2021-09-27 12:05:06,621 DEBUG awx.main.dispatch delivered f491b4d1-e451-4240-8e7c-c07f49559c0f to worker[182] qsize 0
2021-09-27 12:05:06,625 DEBUG awx.main.dispatch task f491b4d1-e451-4240-8e7c-c07f49559c0f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:05:06,627 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:05:08,725 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:05:08,822 DEBUG awx.main.dispatch task f491b4d1-e451-4240-8e7c-c07f49559c0f is finished
2021-09-27 12:05:20,698 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(6dc059ba-8398-457a-b190-d8e352464382, queue=dev-awx-0)
[2021-09-27 12:05:20,698: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(6dc059ba-8398-457a-b190-d8e352464382, queue=dev-awx-0)
2021-09-27 12:05:20,777 DEBUG awx.main.dispatch delivered 6dc059ba-8398-457a-b190-d8e352464382 to worker[183] qsize 0
2021-09-27 12:05:20,782 DEBUG awx.main.dispatch task 6dc059ba-8398-457a-b190-d8e352464382 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 12:05:21,123 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 12:05:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(aeedaae3-ea42-4aa9-9b77-601a330ce011, queue=dev-awx-0)
[2021-09-27 12:05:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(aeedaae3-ea42-4aa9-9b77-601a330ce011, queue=dev-awx-0)
2021-09-27 12:05:21,993 DEBUG awx.main.dispatch delivered aeedaae3-ea42-4aa9-9b77-601a330ce011 to worker[181] qsize 0
2021-09-27 12:05:21,996 DEBUG awx.main.dispatch task aeedaae3-ea42-4aa9-9b77-601a330ce011 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:05:23,098 DEBUG awx.main.dispatch task 6dc059ba-8398-457a-b190-d8e352464382 is finished
2021-09-27 12:05:23,451 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(82438cd2-6cff-4589-b520-5fbf0c1656e4, queue=awx_private_queue)
[2021-09-27 12:05:23,451: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(82438cd2-6cff-4589-b520-5fbf0c1656e4, queue=awx_private_queue)
2021-09-27 12:05:23,477 DEBUG awx.main.dispatch delivered 82438cd2-6cff-4589-b520-5fbf0c1656e4 to worker[180] qsize 0
2021-09-27 12:05:23,480 DEBUG awx.main.dispatch task 82438cd2-6cff-4589-b520-5fbf0c1656e4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:05:25,059 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 12:05:25,071 DEBUG awx.main.dispatch task aeedaae3-ea42-4aa9-9b77-601a330ce011 is finished
2021-09-27 12:05:25,072 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 12:04:24.055275+00:00
2021-09-27 12:05:25,234 DEBUG awx.main.dispatch task 82438cd2-6cff-4589-b520-5fbf0c1656e4 is finished
2021-09-27 12:05:26,596 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(806de18e-2273-4b3f-bef2-38e8d8dcc24e, queue=awx_private_queue)
[2021-09-27 12:05:26,596: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(806de18e-2273-4b3f-bef2-38e8d8dcc24e, queue=awx_private_queue)
2021-09-27 12:05:26,621 DEBUG awx.main.dispatch delivered 806de18e-2273-4b3f-bef2-38e8d8dcc24e to worker[183] qsize 0
2021-09-27 12:05:26,624 DEBUG awx.main.dispatch task 806de18e-2273-4b3f-bef2-38e8d8dcc24e starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:05:26,625 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:05:26,679 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:05:26,732 DEBUG awx.main.dispatch task 806de18e-2273-4b3f-bef2-38e8d8dcc24e is finished
2021-09-27 12:05:46,597 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7d596db1-c496-475f-ab19-cfcbde54d4e9, queue=awx_private_queue)
[2021-09-27 12:05:46,597: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7d596db1-c496-475f-ab19-cfcbde54d4e9, queue=awx_private_queue)
2021-09-27 12:05:46,625 DEBUG awx.main.dispatch delivered 7d596db1-c496-475f-ab19-cfcbde54d4e9 to worker[182] qsize 0
2021-09-27 12:05:46,631 DEBUG awx.main.dispatch task 7d596db1-c496-475f-ab19-cfcbde54d4e9 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:05:46,635 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:05:47,669 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:05:47,673 DEBUG awx.main.dispatch task 7d596db1-c496-475f-ab19-cfcbde54d4e9 is finished
2021-09-27 12:05:53,454 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(7ab0d8dd-fa31-443d-8a03-b28e94a4a70a, queue=awx_private_queue)
[2021-09-27 12:05:53,454: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(7ab0d8dd-fa31-443d-8a03-b28e94a4a70a, queue=awx_private_queue)
2021-09-27 12:05:53,488 DEBUG awx.main.dispatch delivered 7ab0d8dd-fa31-443d-8a03-b28e94a4a70a to worker[182] qsize 0
2021-09-27 12:05:53,493 DEBUG awx.main.dispatch task 7ab0d8dd-fa31-443d-8a03-b28e94a4a70a starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:05:54,513 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:05:54,516 DEBUG awx.main.dispatch task 7ab0d8dd-fa31-443d-8a03-b28e94a4a70a is finished
2021-09-27 12:06:00,003 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(4af7bf87-c81e-4ee3-ae1b-751a67d71c00, queue=awx_private_queue)
[2021-09-27 12:06:00,003: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(4af7bf87-c81e-4ee3-ae1b-751a67d71c00, queue=awx_private_queue)
2021-09-27 12:06:00,031 DEBUG awx.main.dispatch delivered 4af7bf87-c81e-4ee3-ae1b-751a67d71c00 to worker[180] qsize 0
RESULT 2
OKREADY
2021-09-27 12:06:00,035 DEBUG awx.main.dispatch task 4af7bf87-c81e-4ee3-ae1b-751a67d71c00 starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:06:00,040 DEBUG awx.main.dispatch task 4af7bf87-c81e-4ee3-ae1b-751a67d71c00 is finished
2021-09-27 12:06:06,599 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(cae8d34a-df91-477b-9a12-0b6081f784c5, queue=awx_private_queue)
[2021-09-27 12:06:06,599: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(cae8d34a-df91-477b-9a12-0b6081f784c5, queue=awx_private_queue)
2021-09-27 12:06:06,625 DEBUG awx.main.dispatch delivered cae8d34a-df91-477b-9a12-0b6081f784c5 to worker[180] qsize 0
2021-09-27 12:06:06,628 DEBUG awx.main.dispatch task cae8d34a-df91-477b-9a12-0b6081f784c5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:06:06,630 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:06:07,665 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:06:07,669 DEBUG awx.main.dispatch task cae8d34a-df91-477b-9a12-0b6081f784c5 is finished
2021-09-27 12:06:20,700 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(da850c32-e123-4c31-aa50-b0c20de3db16, queue=dev-awx-0)
[2021-09-27 12:06:20,700: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(da850c32-e123-4c31-aa50-b0c20de3db16, queue=dev-awx-0)
2021-09-27 12:06:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(42b4d2b9-21fe-4cba-a9d9-78f6798d509d, queue=dev-awx-0)
[2021-09-27 12:06:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(42b4d2b9-21fe-4cba-a9d9-78f6798d509d, queue=dev-awx-0)
2021-09-27 12:06:23,451 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e1beead1-d5ef-4193-8b12-79889b89ed6b, queue=awx_private_queue)
[2021-09-27 12:06:23,451: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e1beead1-d5ef-4193-8b12-79889b89ed6b, queue=awx_private_queue)
2021-09-27 12:06:26,598 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f08a8d00-7f4e-458e-b86a-0574c90eb46a, queue=awx_private_queue)
[2021-09-27 12:06:26,598: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f08a8d00-7f4e-458e-b86a-0574c90eb46a, queue=awx_private_queue)
2021-09-27 12:06:32,622 DEBUG awx.main.dispatch delivered da850c32-e123-4c31-aa50-b0c20de3db16 to worker[182] qsize 0
2021-09-27 12:06:32,646 DEBUG awx.main.dispatch delivered 42b4d2b9-21fe-4cba-a9d9-78f6798d509d to worker[183] qsize 0
2021-09-27 12:06:32,646 DEBUG awx.main.dispatch task da850c32-e123-4c31-aa50-b0c20de3db16 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 12:06:32,648 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 12:06:32,648 DEBUG awx.main.dispatch delivered e1beead1-d5ef-4193-8b12-79889b89ed6b to worker[181] qsize 0
2021-09-27 12:06:32,650 DEBUG awx.main.dispatch delivered f08a8d00-7f4e-458e-b86a-0574c90eb46a to worker[180] qsize 0
2021-09-27 12:06:32,661 DEBUG awx.main.dispatch task f08a8d00-7f4e-458e-b86a-0574c90eb46a starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:06:32,662 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:06:32,649 DEBUG awx.main.dispatch task 42b4d2b9-21fe-4cba-a9d9-78f6798d509d starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:06:33,681 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:06:33,681 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.cluster_node_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 366, in cluster_node_heartbeat
instance_list = list(Instance.objects.all_non_isolated())
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:06:33,683 DEBUG awx.main.dispatch task f08a8d00-7f4e-458e-b86a-0574c90eb46a is finished
2021-09-27 12:06:33,683 DEBUG awx.main.dispatch task da850c32-e123-4c31-aa50-b0c20de3db16 is finished
2021-09-27 12:06:34,709 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:06:34,712 DEBUG awx.main.dispatch task 42b4d2b9-21fe-4cba-a9d9-78f6798d509d is finished
2021-09-27 12:06:32,651 DEBUG awx.main.dispatch task e1beead1-d5ef-4193-8b12-79889b89ed6b starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:06:36,721 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:06:36,725 DEBUG awx.main.dispatch task e1beead1-d5ef-4193-8b12-79889b89ed6b is finished
2021-09-27 12:06:46,601 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3d669ac1-8e30-493c-8d5a-6132514c2f66, queue=awx_private_queue)
[2021-09-27 12:06:46,601: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3d669ac1-8e30-493c-8d5a-6132514c2f66, queue=awx_private_queue)
2021-09-27 12:06:46,625 DEBUG awx.main.dispatch delivered 3d669ac1-8e30-493c-8d5a-6132514c2f66 to worker[183] qsize 0
2021-09-27 12:06:46,627 DEBUG awx.main.dispatch task 3d669ac1-8e30-493c-8d5a-6132514c2f66 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:06:46,629 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:06:46,660 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:06:46,706 DEBUG awx.main.dispatch task 3d669ac1-8e30-493c-8d5a-6132514c2f66 is finished
2021-09-27 12:06:53,451 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(ed19f5fb-7c04-4655-8d1c-d7ac5f570c85, queue=awx_private_queue)
[2021-09-27 12:06:53,451: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(ed19f5fb-7c04-4655-8d1c-d7ac5f570c85, queue=awx_private_queue)
2021-09-27 12:06:53,475 DEBUG awx.main.dispatch delivered ed19f5fb-7c04-4655-8d1c-d7ac5f570c85 to worker[180] qsize 0
2021-09-27 12:06:53,746 DEBUG awx.main.dispatch task ed19f5fb-7c04-4655-8d1c-d7ac5f570c85 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:06:53,784 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 12:06:56,255 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 12:05:25.060062+00:00
2021-09-27 12:06:58,284 DEBUG awx.main.dispatch task ed19f5fb-7c04-4655-8d1c-d7ac5f570c85 is finished
2021-09-27 12:07:00,002 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(77720c32-3cb4-49ff-8fce-afc7334fd07f, queue=awx_private_queue)
[2021-09-27 12:07:00,002: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(77720c32-3cb4-49ff-8fce-afc7334fd07f, queue=awx_private_queue)
2021-09-27 12:07:00,026 DEBUG awx.main.dispatch delivered 77720c32-3cb4-49ff-8fce-afc7334fd07f to worker[183] qsize 0
RESULT 2
OKREADY
2021-09-27 12:07:00,029 DEBUG awx.main.dispatch task 77720c32-3cb4-49ff-8fce-afc7334fd07f starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:07:00,032 DEBUG awx.main.dispatch task 77720c32-3cb4-49ff-8fce-afc7334fd07f is finished
2021-09-27 12:07:06,601 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(604de841-be7d-4e5b-8031-93e89acbcb92, queue=awx_private_queue)
[2021-09-27 12:07:06,601: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(604de841-be7d-4e5b-8031-93e89acbcb92, queue=awx_private_queue)
2021-09-27 12:07:06,625 DEBUG awx.main.dispatch delivered 604de841-be7d-4e5b-8031-93e89acbcb92 to worker[183] qsize 0
2021-09-27 12:07:06,628 DEBUG awx.main.dispatch task 604de841-be7d-4e5b-8031-93e89acbcb92 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:07:06,630 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:07:20,702 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(ed5d79f2-4641-4f69-8eba-f2a96c98f2b5, queue=dev-awx-0)
[2021-09-27 12:07:20,702: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(ed5d79f2-4641-4f69-8eba-f2a96c98f2b5, queue=dev-awx-0)
2021-09-27 12:07:21,137 DEBUG awx.main.dispatch delivered ed5d79f2-4641-4f69-8eba-f2a96c98f2b5 to worker[181] qsize 0
2021-09-27 12:07:21,140 DEBUG awx.main.dispatch task ed5d79f2-4641-4f69-8eba-f2a96c98f2b5 starting awx.main.tasks.cluster_node_heartbeat(*[])
2021-09-27 12:07:21,142 DEBUG awx.main.tasks Cluster node heartbeat task.
2021-09-27 12:07:21,969 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(019653fc-3bbd-44a3-9982-7a0211400396, queue=dev-awx-0)
[2021-09-27 12:07:21,969: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(019653fc-3bbd-44a3-9982-7a0211400396, queue=dev-awx-0)
2021-09-27 12:07:21,996 DEBUG awx.main.dispatch delivered 019653fc-3bbd-44a3-9982-7a0211400396 to worker[182] qsize 0
2021-09-27 12:07:22,000 DEBUG awx.main.dispatch task 019653fc-3bbd-44a3-9982-7a0211400396 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:07:23,453 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3e0fd479-18d0-4188-913b-9b6ea0582b36, queue=awx_private_queue)
[2021-09-27 12:07:23,453: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3e0fd479-18d0-4188-913b-9b6ea0582b36, queue=awx_private_queue)
2021-09-27 12:07:23,483 DEBUG awx.main.dispatch delivered 3e0fd479-18d0-4188-913b-9b6ea0582b36 to worker[180] qsize 0
2021-09-27 12:07:23,487 DEBUG awx.main.dispatch task 3e0fd479-18d0-4188-913b-9b6ea0582b36 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:07:26,602 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d0adb9ea-d314-4758-aeed-796842bd5351, queue=awx_private_queue)
[2021-09-27 12:07:26,602: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d0adb9ea-d314-4758-aeed-796842bd5351, queue=awx_private_queue)
2021-09-27 12:07:26,642 WARNING awx.main.dispatch scaling up worker pid:192
2021-09-27 12:07:26,650 DEBUG awx.main.dispatch delivered d0adb9ea-d314-4758-aeed-796842bd5351 to worker[192] qsize 0
2021-09-27 12:07:26,658 DEBUG awx.main.dispatch task d0adb9ea-d314-4758-aeed-796842bd5351 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:07:26,662 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:07:30,682 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:07:30,859 DEBUG awx.main.dispatch task 604de841-be7d-4e5b-8031-93e89acbcb92 is finished
2021-09-27 12:07:31,605 DEBUG awx.main.tasks Starting periodic scheduler
2021-09-27 12:07:31,614 DEBUG awx.main.tasks Last scheduler run was: 2021-09-27 12:06:53.784790+00:00
2021-09-27 12:07:31,616 DEBUG awx.main.dispatch task 019653fc-3bbd-44a3-9982-7a0211400396 is finished
2021-09-27 12:07:31,621 DEBUG awx.main.scheduler Starting Scheduler
2021-09-27 12:07:31,651 DEBUG awx.main.dispatch task ed5d79f2-4641-4f69-8eba-f2a96c98f2b5 is finished
2021-09-27 12:07:31,661 DEBUG awx.main.dispatch task 3e0fd479-18d0-4188-913b-9b6ea0582b36 is finished
2021-09-27 12:07:31,751 DEBUG awx.main.dispatch task d0adb9ea-d314-4758-aeed-796842bd5351 is finished
2021-09-27 12:07:46,606 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d19d1600-4354-46a5-b6dc-4854e5e23ade, queue=awx_private_queue)
[2021-09-27 12:07:46,606: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d19d1600-4354-46a5-b6dc-4854e5e23ade, queue=awx_private_queue)
2021-09-27 12:07:46,633 DEBUG awx.main.dispatch delivered d19d1600-4354-46a5-b6dc-4854e5e23ade to worker[182] qsize 0
2021-09-27 12:07:46,637 DEBUG awx.main.dispatch task d19d1600-4354-46a5-b6dc-4854e5e23ade starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:07:46,639 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:07:47,665 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:07:47,669 DEBUG awx.main.dispatch task d19d1600-4354-46a5-b6dc-4854e5e23ade is finished
2021-09-27 12:07:53,452 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(3d9ff02d-c559-459e-b9a7-d100b2ec6e0c, queue=awx_private_queue)
[2021-09-27 12:07:53,452: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(3d9ff02d-c559-459e-b9a7-d100b2ec6e0c, queue=awx_private_queue)
2021-09-27 12:07:54,506 DEBUG awx.main.dispatch delivered 3d9ff02d-c559-459e-b9a7-d100b2ec6e0c to worker[180] qsize 0
2021-09-27 12:07:55,546 DEBUG awx.main.dispatch task 3d9ff02d-c559-459e-b9a7-d100b2ec6e0c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:07:57,585 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:07:58,612 DEBUG awx.main.dispatch task 3d9ff02d-c559-459e-b9a7-d100b2ec6e0c is finished
2021-09-27 12:08:00,001 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(66200607-f7ef-4ca3-8cc8-14e62cbeb4b4, queue=awx_private_queue)
[2021-09-27 12:08:00,001: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(66200607-f7ef-4ca3-8cc8-14e62cbeb4b4, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 12:08:01,034 DEBUG awx.main.dispatch delivered 66200607-f7ef-4ca3-8cc8-14e62cbeb4b4 to worker[181] qsize 0
2021-09-27 12:08:02,070 DEBUG awx.main.dispatch task 66200607-f7ef-4ca3-8cc8-14e62cbeb4b4 starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:08:04,114 DEBUG awx.main.dispatch task 66200607-f7ef-4ca3-8cc8-14e62cbeb4b4 is finished
2021-09-27 12:08:06,607 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(92620691-b5cc-4ecd-983a-9620dbc5f108, queue=awx_private_queue)
[2021-09-27 12:08:06,607: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(92620691-b5cc-4ecd-983a-9620dbc5f108, queue=awx_private_queue)
2021-09-27 12:08:07,658 DEBUG awx.main.dispatch delivered 92620691-b5cc-4ecd-983a-9620dbc5f108 to worker[181] qsize 0
2021-09-27 12:08:08,692 DEBUG awx.main.dispatch task 92620691-b5cc-4ecd-983a-9620dbc5f108 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:08:09,715 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:08:10,737 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:08:11,764 DEBUG awx.main.dispatch task 92620691-b5cc-4ecd-983a-9620dbc5f108 is finished
2021-09-27 12:08:20,702 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(5064dfa5-db07-47bf-abcb-94e5f4f55d2e, queue=dev-awx-0)
[2021-09-27 12:08:20,702: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(5064dfa5-db07-47bf-abcb-94e5f4f55d2e, queue=dev-awx-0)
2021-09-27 12:08:21,737 WARNING awx.main.dispatch scaling down worker pid:180
2021-09-27 12:08:21,967 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(0721a0a0-22ea-4e2d-a272-82186261f32f, queue=dev-awx-0)
[2021-09-27 12:08:21,967: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(0721a0a0-22ea-4e2d-a272-82186261f32f, queue=dev-awx-0)
2021-09-27 12:08:22,777 WARNING awx.main.dispatch worker exiting gracefully pid:180
2021-09-27 12:08:23,797 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 123, in register
with advisory_lock('instance_registration_%s' % hostname):
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:08:24,821 DEBUG awx.main.dispatch delivered 0721a0a0-22ea-4e2d-a272-82186261f32f to worker[182] qsize 0
2021-09-27 12:08:25,846 DEBUG awx.main.dispatch task 0721a0a0-22ea-4e2d-a272-82186261f32f starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:08:27,889 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:08:28,916 DEBUG awx.main.dispatch task 0721a0a0-22ea-4e2d-a272-82186261f32f is finished
2021-09-27 12:08:23,452 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2944ebb7-da83-442a-8f2a-3e7ec1cfb102, queue=awx_private_queue)
[2021-09-27 12:08:23,452: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2944ebb7-da83-442a-8f2a-3e7ec1cfb102, queue=awx_private_queue)
2021-09-27 12:08:38,732 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(498536ed-8d26-4cbc-84e0-b3f40ef2df65, queue=awx_private_queue)
2021-09-27 12:08:38,731 DEBUG awx.main.dispatch delivered 2944ebb7-da83-442a-8f2a-3e7ec1cfb102 to worker[181] qsize 0
[2021-09-27 12:08:38,732: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(498536ed-8d26-4cbc-84e0-b3f40ef2df65, queue=awx_private_queue)
2021-09-27 12:08:39,764 DEBUG awx.main.dispatch task 2944ebb7-da83-442a-8f2a-3e7ec1cfb102 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:08:41,809 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:08:42,835 DEBUG awx.main.dispatch task 2944ebb7-da83-442a-8f2a-3e7ec1cfb102 is finished
2021-09-27 12:08:39,784 DEBUG awx.main.dispatch delivered 498536ed-8d26-4cbc-84e0-b3f40ef2df65 to worker[192] qsize 0
2021-09-27 12:08:46,904 DEBUG awx.main.dispatch task 498536ed-8d26-4cbc-84e0-b3f40ef2df65 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:08:47,936 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:08:48,949 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:08:49,975 DEBUG awx.main.dispatch task 498536ed-8d26-4cbc-84e0-b3f40ef2df65 is finished
2021-09-27 12:08:58,747 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(b7cd07bd-484b-45a1-b9e2-bfd9ac75bb37, queue=awx_private_queue)
[2021-09-27 12:08:58,747: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(b7cd07bd-484b-45a1-b9e2-bfd9ac75bb37, queue=awx_private_queue)
2021-09-27 12:08:59,791 DEBUG awx.main.dispatch delivered b7cd07bd-484b-45a1-b9e2-bfd9ac75bb37 to worker[183] qsize 0
RESULT 2
OKREADY
2021-09-27 12:09:00,002 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(4217278d-d087-4afa-9998-f22cbe384104, queue=awx_private_queue)
[2021-09-27 12:09:00,002: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(4217278d-d087-4afa-9998-f22cbe384104, queue=awx_private_queue)
2021-09-27 12:09:00,823 DEBUG awx.main.dispatch task b7cd07bd-484b-45a1-b9e2-bfd9ac75bb37 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:09:01,843 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:09:02,869 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:09:03,892 DEBUG awx.main.dispatch task b7cd07bd-484b-45a1-b9e2-bfd9ac75bb37 is finished
2021-09-27 12:09:01,033 DEBUG awx.main.dispatch delivered 4217278d-d087-4afa-9998-f22cbe384104 to worker[181] qsize 0
2021-09-27 12:09:01,036 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(082f85d8-27b2-4010-8c18-260122e26c82, queue=awx_private_queue)
[2021-09-27 12:09:01,036: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(082f85d8-27b2-4010-8c18-260122e26c82, queue=awx_private_queue)
2021-09-27 12:09:08,148 DEBUG awx.main.dispatch task 4217278d-d087-4afa-9998-f22cbe384104 starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:09:10,194 DEBUG awx.main.dispatch task 4217278d-d087-4afa-9998-f22cbe384104 is finished
2021-09-27 12:09:08,172 DEBUG awx.main.dispatch delivered 082f85d8-27b2-4010-8c18-260122e26c82 to worker[182] qsize 0
2021-09-27 12:09:15,316 DEBUG awx.main.dispatch task 082f85d8-27b2-4010-8c18-260122e26c82 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:09:17,361 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:09:18,388 DEBUG awx.main.dispatch task 082f85d8-27b2-4010-8c18-260122e26c82 is finished
2021-09-27 12:09:18,746 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2748cd80-5a7c-4884-ad26-426380d5a38f, queue=awx_private_queue)
[2021-09-27 12:09:18,746: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2748cd80-5a7c-4884-ad26-426380d5a38f, queue=awx_private_queue)
2021-09-27 12:09:19,787 DEBUG awx.main.dispatch delivered 2748cd80-5a7c-4884-ad26-426380d5a38f to worker[182] qsize 0
2021-09-27 12:09:20,821 DEBUG awx.main.dispatch task 2748cd80-5a7c-4884-ad26-426380d5a38f starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:09:21,843 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:09:22,865 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:09:23,891 DEBUG awx.main.dispatch task 2748cd80-5a7c-4884-ad26-426380d5a38f is finished
2021-09-27 12:09:20,702 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(985ba38f-bb09-4ff5-805b-2d9026cac5c1, queue=dev-awx-0)
[2021-09-27 12:09:20,702: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(985ba38f-bb09-4ff5-805b-2d9026cac5c1, queue=dev-awx-0)
2021-09-27 12:09:28,913 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 123, in register
with advisory_lock('instance_registration_%s' % hostname):
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:09:27,884 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(2e754899-69a5-42cc-ae2c-594f6f669d14, queue=dev-awx-0)
[2021-09-27 12:09:27,884: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(2e754899-69a5-42cc-ae2c-594f6f669d14, queue=dev-awx-0)
2021-09-27 12:09:30,952 DEBUG awx.main.dispatch delivered 2e754899-69a5-42cc-ae2c-594f6f669d14 to worker[183] qsize 0
2021-09-27 12:09:31,988 DEBUG awx.main.dispatch task 2e754899-69a5-42cc-ae2c-594f6f669d14 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:09:34,033 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:09:35,060 DEBUG awx.main.dispatch task 2e754899-69a5-42cc-ae2c-594f6f669d14 is finished
2021-09-27 12:09:31,036 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f60ac661-6afe-4939-97b9-0a33e92830cc, queue=awx_private_queue)
[2021-09-27 12:09:31,036: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f60ac661-6afe-4939-97b9-0a33e92830cc, queue=awx_private_queue)
2021-09-27 12:09:38,122 DEBUG awx.main.dispatch delivered f60ac661-6afe-4939-97b9-0a33e92830cc to worker[181] qsize 0
2021-09-27 12:09:38,745 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(711a5a8d-e26b-421e-b127-860e1fef9a42, queue=awx_private_queue)
[2021-09-27 12:09:38,745: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(711a5a8d-e26b-421e-b127-860e1fef9a42, queue=awx_private_queue)
2021-09-27 12:09:39,784 DEBUG awx.main.dispatch delivered 711a5a8d-e26b-421e-b127-860e1fef9a42 to worker[183] qsize 0
2021-09-27 12:09:40,820 DEBUG awx.main.dispatch task 711a5a8d-e26b-421e-b127-860e1fef9a42 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:09:41,843 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:09:42,865 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:09:43,891 DEBUG awx.main.dispatch task 711a5a8d-e26b-421e-b127-860e1fef9a42 is finished
2021-09-27 12:09:39,156 DEBUG awx.main.dispatch task f60ac661-6afe-4939-97b9-0a33e92830cc starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:09:47,313 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:09:48,340 DEBUG awx.main.dispatch task f60ac661-6afe-4939-97b9-0a33e92830cc is finished
2021-09-27 12:09:58,755 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(998dadde-5937-487d-8dad-569af51f5be1, queue=awx_private_queue)
[2021-09-27 12:09:58,755: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(998dadde-5937-487d-8dad-569af51f5be1, queue=awx_private_queue)
2021-09-27 12:09:59,786 DEBUG awx.main.dispatch delivered 998dadde-5937-487d-8dad-569af51f5be1 to worker[192] qsize 0
RESULT 2
OKREADY
2021-09-27 12:10:00,001 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(9797c65e-57a2-4eff-b181-01d5406f580b, queue=awx_private_queue)
[2021-09-27 12:10:00,001: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(9797c65e-57a2-4eff-b181-01d5406f580b, queue=awx_private_queue)
2021-09-27 12:10:00,821 DEBUG awx.main.dispatch task 998dadde-5937-487d-8dad-569af51f5be1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:10:01,843 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:10:02,869 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:10:03,891 DEBUG awx.main.dispatch task 998dadde-5937-487d-8dad-569af51f5be1 is finished
2021-09-27 12:10:01,036 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(f91cd67c-5ce8-4a19-aa5d-bad85b3e781d, queue=awx_private_queue)
2021-09-27 12:10:01,032 DEBUG awx.main.dispatch delivered 9797c65e-57a2-4eff-b181-01d5406f580b to worker[181] qsize 0
[2021-09-27 12:10:01,036: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(f91cd67c-5ce8-4a19-aa5d-bad85b3e781d, queue=awx_private_queue)
2021-09-27 12:10:08,308 DEBUG awx.main.dispatch task 9797c65e-57a2-4eff-b181-01d5406f580b starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:10:10,354 DEBUG awx.main.dispatch task 9797c65e-57a2-4eff-b181-01d5406f580b is finished
2021-09-27 12:10:08,330 DEBUG awx.main.dispatch delivered f91cd67c-5ce8-4a19-aa5d-bad85b3e781d to worker[182] qsize 0
2021-09-27 12:10:15,478 DEBUG awx.main.dispatch task f91cd67c-5ce8-4a19-aa5d-bad85b3e781d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:10:17,521 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:10:18,758 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(d37a9358-0093-434f-ab11-b3712394b824, queue=awx_private_queue)
[2021-09-27 12:10:18,758: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(d37a9358-0093-434f-ab11-b3712394b824, queue=awx_private_queue)
2021-09-27 12:10:19,823 DEBUG awx.main.dispatch delivered d37a9358-0093-434f-ab11-b3712394b824 to worker[183] qsize 0
2021-09-27 12:10:20,854 DEBUG awx.main.dispatch task d37a9358-0093-434f-ab11-b3712394b824 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:10:21,876 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:10:22,897 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:10:23,928 DEBUG awx.main.dispatch task d37a9358-0093-434f-ab11-b3712394b824 is finished
2021-09-27 12:10:20,703 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(e25f673e-5004-4259-a8e8-da74f24c34e2, queue=dev-awx-0)
[2021-09-27 12:10:20,703: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(e25f673e-5004-4259-a8e8-da74f24c34e2, queue=dev-awx-0)
2021-09-27 12:10:28,817 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 123, in register
with advisory_lock('instance_registration_%s' % hostname):
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:10:27,886 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(3665821e-ee8e-41fb-906f-7299d989dac6, queue=dev-awx-0)
[2021-09-27 12:10:27,886: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(3665821e-ee8e-41fb-906f-7299d989dac6, queue=dev-awx-0)
2021-09-27 12:10:30,954 DEBUG awx.main.dispatch delivered 3665821e-ee8e-41fb-906f-7299d989dac6 to worker[192] qsize 0
2021-09-27 12:10:31,989 DEBUG awx.main.dispatch task 3665821e-ee8e-41fb-906f-7299d989dac6 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:10:18,552 DEBUG awx.main.dispatch task f91cd67c-5ce8-4a19-aa5d-bad85b3e781d is finished
2021-09-27 12:10:36,049 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:10:37,079 DEBUG awx.main.dispatch task 3665821e-ee8e-41fb-906f-7299d989dac6 is finished
2021-09-27 12:10:31,036 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(889cef36-cc5d-452a-b3b4-deba3b1257f4, queue=awx_private_queue)
[2021-09-27 12:10:31,036: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(889cef36-cc5d-452a-b3b4-deba3b1257f4, queue=awx_private_queue)
2021-09-27 12:10:46,473 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(531b507a-91a6-4444-9432-bbd1c2772406, queue=awx_private_queue)
[2021-09-27 12:10:46,473: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(531b507a-91a6-4444-9432-bbd1c2772406, queue=awx_private_queue)
2021-09-27 12:10:46,472 DEBUG awx.main.dispatch delivered 889cef36-cc5d-452a-b3b4-deba3b1257f4 to worker[192] qsize 0
2021-09-27 12:10:47,509 DEBUG awx.main.dispatch task 889cef36-cc5d-452a-b3b4-deba3b1257f4 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:10:47,526 DEBUG awx.main.dispatch delivered 531b507a-91a6-4444-9432-bbd1c2772406 to worker[183] qsize 0
2021-09-27 12:10:50,581 DEBUG awx.main.dispatch task 531b507a-91a6-4444-9432-bbd1c2772406 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:10:51,603 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:10:49,553 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:10:52,601 DEBUG awx.main.dispatch task 889cef36-cc5d-452a-b3b4-deba3b1257f4 is finished
2021-09-27 12:10:54,641 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:10:55,670 DEBUG awx.main.dispatch task 531b507a-91a6-4444-9432-bbd1c2772406 is finished
2021-09-27 12:11:00,002 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(2384c1cc-fa2e-4e05-9a84-716ee781b7e9, queue=awx_private_queue)
[2021-09-27 12:11:00,002: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(2384c1cc-fa2e-4e05-9a84-716ee781b7e9, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 12:11:01,068 DEBUG awx.main.dispatch delivered 2384c1cc-fa2e-4e05-9a84-716ee781b7e9 to worker[192] qsize 0
2021-09-27 12:11:02,102 DEBUG awx.main.dispatch task 2384c1cc-fa2e-4e05-9a84-716ee781b7e9 starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:11:04,146 DEBUG awx.main.dispatch task 2384c1cc-fa2e-4e05-9a84-716ee781b7e9 is finished
2021-09-27 12:11:06,474 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(7391881f-bca7-4438-9fe8-f09867574e6c, queue=awx_private_queue)
[2021-09-27 12:11:06,474: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(7391881f-bca7-4438-9fe8-f09867574e6c, queue=awx_private_queue)
2021-09-27 12:11:07,532 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(983c59d4-d0e6-495e-af5a-8323c428fa1f, queue=awx_private_queue)
[2021-09-27 12:11:07,532: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(983c59d4-d0e6-495e-af5a-8323c428fa1f, queue=awx_private_queue)
2021-09-27 12:11:07,530 DEBUG awx.main.dispatch delivered 7391881f-bca7-4438-9fe8-f09867574e6c to worker[183] qsize 0
2021-09-27 12:11:10,584 DEBUG awx.main.dispatch task 7391881f-bca7-4438-9fe8-f09867574e6c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:11:11,604 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:11:10,585 DEBUG awx.main.dispatch delivered 983c59d4-d0e6-495e-af5a-8323c428fa1f to worker[192] qsize 0
2021-09-27 12:11:13,620 DEBUG awx.main.dispatch task 983c59d4-d0e6-495e-af5a-8323c428fa1f starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:11:12,625 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:11:15,667 DEBUG awx.main.dispatch task 7391881f-bca7-4438-9fe8-f09867574e6c is finished
2021-09-27 12:11:17,681 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:11:18,708 DEBUG awx.main.dispatch task 983c59d4-d0e6-495e-af5a-8323c428fa1f is finished
2021-09-27 12:11:26,474 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(051db826-77c1-43a7-85ba-dd6852e3133c, queue=awx_private_queue)
[2021-09-27 12:11:26,474: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(051db826-77c1-43a7-85ba-dd6852e3133c, queue=awx_private_queue)
2021-09-27 12:11:27,530 DEBUG awx.main.dispatch delivered 051db826-77c1-43a7-85ba-dd6852e3133c to worker[192] qsize 0
2021-09-27 12:11:27,531 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(248ee275-a491-4681-b727-739ad8beda61, queue=dev-awx-0)
[2021-09-27 12:11:27,531: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(248ee275-a491-4681-b727-739ad8beda61, queue=dev-awx-0)
2021-09-27 12:11:28,563 DEBUG awx.main.dispatch task 051db826-77c1-43a7-85ba-dd6852e3133c starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:11:29,587 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:11:31,633 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 123, in register
with advisory_lock('instance_registration_%s' % hostname):
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:11:30,609 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:11:33,652 DEBUG awx.main.dispatch task 051db826-77c1-43a7-85ba-dd6852e3133c is finished
2021-09-27 12:11:28,589 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(b64d0be3-b3d9-447e-8405-b7ca8154de52, queue=dev-awx-0)
[2021-09-27 12:11:28,589: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(b64d0be3-b3d9-447e-8405-b7ca8154de52, queue=dev-awx-0)
2021-09-27 12:11:35,881 DEBUG awx.main.dispatch delivered b64d0be3-b3d9-447e-8405-b7ca8154de52 to worker[183] qsize 0
2021-09-27 12:11:37,532 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(6bbad93b-fbe4-4752-a491-2a8ce0b9b4a9, queue=awx_private_queue)
[2021-09-27 12:11:37,532: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(6bbad93b-fbe4-4752-a491-2a8ce0b9b4a9, queue=awx_private_queue)
2021-09-27 12:11:38,565 DEBUG awx.main.dispatch delivered 6bbad93b-fbe4-4752-a491-2a8ce0b9b4a9 to worker[182] qsize 0
2021-09-27 12:11:39,604 DEBUG awx.main.dispatch task 6bbad93b-fbe4-4752-a491-2a8ce0b9b4a9 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:11:41,649 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:11:42,675 DEBUG awx.main.dispatch task 6bbad93b-fbe4-4752-a491-2a8ce0b9b4a9 is finished
2021-09-27 12:11:46,474 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(865b5463-1920-437d-a0c6-e27ffb6eabea, queue=awx_private_queue)
[2021-09-27 12:11:46,474: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(865b5463-1920-437d-a0c6-e27ffb6eabea, queue=awx_private_queue)
2021-09-27 12:11:47,530 DEBUG awx.main.dispatch delivered 865b5463-1920-437d-a0c6-e27ffb6eabea to worker[181] qsize 0
2021-09-27 12:11:48,565 DEBUG awx.main.dispatch task 865b5463-1920-437d-a0c6-e27ffb6eabea starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:11:49,588 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:11:50,609 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:11:51,638 DEBUG awx.main.dispatch task 865b5463-1920-437d-a0c6-e27ffb6eabea is finished
2021-09-27 12:12:00,006 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(2f213e0c-599b-4d1a-b8d5-a2ed9a2368b3, queue=awx_private_queue)
[2021-09-27 12:12:00,006: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(2f213e0c-599b-4d1a-b8d5-a2ed9a2368b3, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 12:12:01,070 DEBUG awx.main.dispatch delivered 2f213e0c-599b-4d1a-b8d5-a2ed9a2368b3 to worker[182] qsize 0
2021-09-27 12:12:02,100 DEBUG awx.main.dispatch task 2f213e0c-599b-4d1a-b8d5-a2ed9a2368b3 starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:12:04,147 DEBUG awx.main.dispatch task 2f213e0c-599b-4d1a-b8d5-a2ed9a2368b3 is finished
2021-09-27 12:12:06,474 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(da06f2ee-358f-4733-a721-ff43253755b1, queue=awx_private_queue)
[2021-09-27 12:12:06,474: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(da06f2ee-358f-4733-a721-ff43253755b1, queue=awx_private_queue)
2021-09-27 12:11:36,916 DEBUG awx.main.dispatch task b64d0be3-b3d9-447e-8405-b7ca8154de52 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:12:09,393 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:12:10,419 DEBUG awx.main.dispatch task b64d0be3-b3d9-447e-8405-b7ca8154de52 is finished
2021-09-27 12:12:07,528 DEBUG awx.main.dispatch delivered da06f2ee-358f-4733-a721-ff43253755b1 to worker[192] qsize 0
2021-09-27 12:12:07,532 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(e7248fda-bdac-48a0-af66-c1ff2e48184d, queue=awx_private_queue)
[2021-09-27 12:12:07,532: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(e7248fda-bdac-48a0-af66-c1ff2e48184d, queue=awx_private_queue)
2021-09-27 12:12:14,793 DEBUG awx.main.dispatch delivered e7248fda-bdac-48a0-af66-c1ff2e48184d to worker[182] qsize 0
2021-09-27 12:12:14,776 DEBUG awx.main.dispatch task da06f2ee-358f-4733-a721-ff43253755b1 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:12:15,827 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:12:16,849 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:12:15,827 DEBUG awx.main.dispatch task e7248fda-bdac-48a0-af66-c1ff2e48184d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:12:19,889 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:12:20,916 DEBUG awx.main.dispatch task e7248fda-bdac-48a0-af66-c1ff2e48184d is finished
2021-09-27 12:12:17,876 DEBUG awx.main.dispatch task da06f2ee-358f-4733-a721-ff43253755b1 is finished
2021-09-27 12:12:26,477 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(5674fd65-2711-4c25-8297-0be8929588d2, queue=awx_private_queue)
[2021-09-27 12:12:26,477: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(5674fd65-2711-4c25-8297-0be8929588d2, queue=awx_private_queue)
2021-09-27 12:12:27,531 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(7fc0c514-2219-4333-a0f8-3e8a9b4954b4, queue=dev-awx-0)
[2021-09-27 12:12:27,531: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(7fc0c514-2219-4333-a0f8-3e8a9b4954b4, queue=dev-awx-0)
2021-09-27 12:12:27,530 DEBUG awx.main.dispatch delivered 5674fd65-2711-4c25-8297-0be8929588d2 to worker[192] qsize 0
2021-09-27 12:12:28,564 DEBUG awx.main.dispatch task 5674fd65-2711-4c25-8297-0be8929588d2 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:12:29,587 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:12:30,609 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:12:31,636 DEBUG awx.main.dispatch task 5674fd65-2711-4c25-8297-0be8929588d2 is finished
2021-09-27 12:12:28,590 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(ac27f91d-aad1-47e9-a72d-ba672fe907ef, queue=dev-awx-0)
[2021-09-27 12:12:28,590: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(ac27f91d-aad1-47e9-a72d-ba672fe907ef, queue=dev-awx-0)
2021-09-27 12:12:35,761 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 123, in register
with advisory_lock('instance_registration_%s' % hostname):
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:12:37,533 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(97a439a1-0be0-496a-b7a2-d84c1f0d7b85, queue=awx_private_queue)
[2021-09-27 12:12:37,533: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(97a439a1-0be0-496a-b7a2-d84c1f0d7b85, queue=awx_private_queue)
2021-09-27 12:12:36,790 DEBUG awx.main.dispatch delivered ac27f91d-aad1-47e9-a72d-ba672fe907ef to worker[183] qsize 0
2021-09-27 12:12:39,831 DEBUG awx.main.dispatch delivered 97a439a1-0be0-496a-b7a2-d84c1f0d7b85 to worker[181] qsize 0
2021-09-27 12:12:40,853 DEBUG awx.main.dispatch task 97a439a1-0be0-496a-b7a2-d84c1f0d7b85 starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:12:39,831 DEBUG awx.main.dispatch task ac27f91d-aad1-47e9-a72d-ba672fe907ef starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:12:43,889 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:12:44,915 DEBUG awx.main.dispatch task ac27f91d-aad1-47e9-a72d-ba672fe907ef is finished
2021-09-27 12:12:46,477 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(0b3c531e-9117-45f9-a457-8a9584da5fb0, queue=awx_private_queue)
[2021-09-27 12:12:46,477: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(0b3c531e-9117-45f9-a457-8a9584da5fb0, queue=awx_private_queue)
2021-09-27 12:12:47,529 DEBUG awx.main.dispatch delivered 0b3c531e-9117-45f9-a457-8a9584da5fb0 to worker[192] qsize 0
2021-09-27 12:12:49,073 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:12:50,100 DEBUG awx.main.dispatch task 97a439a1-0be0-496a-b7a2-d84c1f0d7b85 is finished
2021-09-27 12:12:48,564 DEBUG awx.main.dispatch task 0b3c531e-9117-45f9-a457-8a9584da5fb0 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:12:55,731 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:12:56,753 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:12:57,780 DEBUG awx.main.dispatch task 0b3c531e-9117-45f9-a457-8a9584da5fb0 is finished
2021-09-27 12:13:00,001 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(c95f8866-cb62-4e6a-8bb4-61598d259f3d, queue=awx_private_queue)
[2021-09-27 12:13:00,001: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(c95f8866-cb62-4e6a-8bb4-61598d259f3d, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 12:13:01,033 DEBUG awx.main.dispatch delivered c95f8866-cb62-4e6a-8bb4-61598d259f3d to worker[182] qsize 0
2021-09-27 12:13:02,071 DEBUG awx.main.dispatch task c95f8866-cb62-4e6a-8bb4-61598d259f3d starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:13:04,114 DEBUG awx.main.dispatch task c95f8866-cb62-4e6a-8bb4-61598d259f3d is finished
2021-09-27 12:13:06,476 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(df710d41-64ed-42d9-addb-2082d710b406, queue=awx_private_queue)
[2021-09-27 12:13:06,476: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(df710d41-64ed-42d9-addb-2082d710b406, queue=awx_private_queue)
2021-09-27 12:13:07,529 DEBUG awx.main.dispatch delivered df710d41-64ed-42d9-addb-2082d710b406 to worker[192] qsize 0
2021-09-27 12:13:07,532 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(2774452f-1db3-48c6-a9a0-e1b3b184665e, queue=awx_private_queue)
[2021-09-27 12:13:07,532: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(2774452f-1db3-48c6-a9a0-e1b3b184665e, queue=awx_private_queue)
2021-09-27 12:13:52,573 DEBUG awx.main.dispatch delivered 2774452f-1db3-48c6-a9a0-e1b3b184665e to worker[182] qsize 0
2021-09-27 12:13:52,577 DEBUG awx.main.dispatch task 2774452f-1db3-48c6-a9a0-e1b3b184665e starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:13:53,585 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:13:53,589 DEBUG awx.main.dispatch task 2774452f-1db3-48c6-a9a0-e1b3b184665e is finished
2021-09-27 12:13:52,577 DEBUG awx.main.dispatch publish awx.main.tasks.awx_isolated_heartbeat(e3fdaa29-a92d-4312-813f-15e2d096673d, queue=dev-awx-0)
[2021-09-27 12:13:52,577: DEBUG/Process-1] publish awx.main.tasks.awx_isolated_heartbeat(e3fdaa29-a92d-4312-813f-15e2d096673d, queue=dev-awx-0)
2021-09-27 12:13:53,642 DEBUG awx.main.dispatch delivered e3fdaa29-a92d-4312-813f-15e2d096673d to worker[182] qsize 0
2021-09-27 12:13:53,644 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(3a4b119a-61a5-4afe-9943-e34ff09d9f46, queue=awx_private_queue)
[2021-09-27 12:13:53,644: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(3a4b119a-61a5-4afe-9943-e34ff09d9f46, queue=awx_private_queue)
2021-09-27 12:13:53,645 DEBUG awx.main.dispatch task e3fdaa29-a92d-4312-813f-15e2d096673d starting awx.main.tasks.awx_isolated_heartbeat(*[])
2021-09-27 12:13:53,649 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
2021-09-27 12:13:53,670 DEBUG awx.main.dispatch delivered 3a4b119a-61a5-4afe-9943-e34ff09d9f46 to worker[183] qsize 0
2021-09-27 12:13:53,671 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(bf5b5037-209a-432b-bf49-89454516aaf0, queue=dev-awx-0)
[2021-09-27 12:13:53,671: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(bf5b5037-209a-432b-bf49-89454516aaf0, queue=dev-awx-0)
2021-09-27 12:13:53,690 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(4cd94ca5-8cae-41e5-8d3a-4a7efaab72c0, queue=dev-awx-0)
[2021-09-27 12:13:53,690: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(4cd94ca5-8cae-41e5-8d3a-4a7efaab72c0, queue=dev-awx-0)
2021-09-27 12:13:54,705 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_isolated_heartbeat(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 469, in awx_isolated_heartbeat
with transaction.atomic():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/transaction.py", line 175, in __enter__
if not connection.get_autocommit():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 379, in get_autocommit
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:13:54,710 DEBUG awx.main.dispatch task e3fdaa29-a92d-4312-813f-15e2d096673d is finished
2021-09-27 12:13:53,673 DEBUG awx.main.dispatch task 3a4b119a-61a5-4afe-9943-e34ff09d9f46 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:13:56,723 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:13:56,725 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 400, in write
self.cleanup()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 385, in cleanup
reaper.reap(excluded_uuids=running_uuids)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 132, in get_or_register
return self.register()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/managers.py", line 123, in register
with advisory_lock('instance_registration_%s' % hostname):
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:13:56,729 DEBUG awx.main.dispatch delivered 4cd94ca5-8cae-41e5-8d3a-4a7efaab72c0 to worker[182] qsize 0
2021-09-27 12:13:56,732 DEBUG awx.main.dispatch task 4cd94ca5-8cae-41e5-8d3a-4a7efaab72c0 starting awx.main.tasks.awx_k8s_reaper(*[])
2021-09-27 12:13:57,745 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_k8s_reaper(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 436, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(credential__isnull=False).iterator():
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 341, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1096, in execute_sql
cursor = self.connection.chunked_cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 233, in chunked_cursor
self._named_cursor_idx,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:13:57,749 DEBUG awx.main.dispatch task 4cd94ca5-8cae-41e5-8d3a-4a7efaab72c0 is finished
2021-09-27 12:13:59,761 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:13:59,764 DEBUG awx.main.dispatch task 3a4b119a-61a5-4afe-9943-e34ff09d9f46 is finished
2021-09-27 12:14:00,001 DEBUG awx.main.dispatch publish awx.main.tasks.gather_analytics(b7cbfbc5-b63b-40c1-a793-85edfdb7ab76, queue=awx_private_queue)
[2021-09-27 12:14:00,001: DEBUG/Process-1] publish awx.main.tasks.gather_analytics(b7cbfbc5-b63b-40c1-a793-85edfdb7ab76, queue=awx_private_queue)
RESULT 2
OKREADY
2021-09-27 12:14:00,023 DEBUG awx.main.dispatch delivered b7cbfbc5-b63b-40c1-a793-85edfdb7ab76 to worker[182] qsize 0
2021-09-27 12:14:00,027 DEBUG awx.main.dispatch task b7cbfbc5-b63b-40c1-a793-85edfdb7ab76 starting awx.main.tasks.gather_analytics(*[])
2021-09-27 12:14:01,042 DEBUG awx.main.dispatch task b7cbfbc5-b63b-40c1-a793-85edfdb7ab76 is finished
2021-09-27 12:13:08,563 DEBUG awx.main.dispatch task df710d41-64ed-42d9-addb-2082d710b406 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:14:09,103 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:14:13,650 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(f288fcc3-80df-4b4b-a8ec-5528fd161ccb, queue=awx_private_queue)
[2021-09-27 12:14:13,650: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(f288fcc3-80df-4b4b-a8ec-5528fd161ccb, queue=awx_private_queue)
2021-09-27 12:14:13,676 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(10055204-5094-406d-bcca-ae1aa29d9c3c, queue=awx_private_queue)
[2021-09-27 12:14:13,676: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(10055204-5094-406d-bcca-ae1aa29d9c3c, queue=awx_private_queue)
2021-09-27 12:14:13,676 DEBUG awx.main.dispatch delivered f288fcc3-80df-4b4b-a8ec-5528fd161ccb to worker[181] qsize 0
2021-09-27 12:14:14,712 DEBUG awx.main.dispatch delivered 10055204-5094-406d-bcca-ae1aa29d9c3c to worker[183] qsize 0
2021-09-27 12:14:14,715 DEBUG awx.main.dispatch task 10055204-5094-406d-bcca-ae1aa29d9c3c starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:14:14,713 DEBUG awx.main.dispatch task f288fcc3-80df-4b4b-a8ec-5528fd161ccb starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:14:15,731 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:14:15,729 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:14:15,733 DEBUG awx.main.dispatch task 10055204-5094-406d-bcca-ae1aa29d9c3c is finished
2021-09-27 12:14:16,757 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:14:16,760 DEBUG awx.main.dispatch task f288fcc3-80df-4b4b-a8ec-5528fd161ccb is finished
2021-09-27 12:14:33,658 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(2227ed6e-e617-4ec7-9feb-36067a0e27dc, queue=awx_private_queue)
[2021-09-27 12:14:33,658: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(2227ed6e-e617-4ec7-9feb-36067a0e27dc, queue=awx_private_queue)
2021-09-27 12:14:33,676 DEBUG awx.main.dispatch delivered 2227ed6e-e617-4ec7-9feb-36067a0e27dc to worker[181] qsize 0
2021-09-27 12:14:33,678 DEBUG awx.main.dispatch task 2227ed6e-e617-4ec7-9feb-36067a0e27dc starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:14:33,679 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:14:34,705 ERROR awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/tasks.py", line 15, in run_task_manager
TaskManager().schedule()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/scheduler/task_manager.py", line 45, in __init__
for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:14:34,708 DEBUG awx.main.dispatch task 2227ed6e-e617-4ec7-9feb-36067a0e27dc is finished
2021-09-27 12:14:43,677 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(01998887-0288-42da-96af-c271213ce87d, queue=awx_private_queue)
[2021-09-27 12:14:43,677: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(01998887-0288-42da-96af-c271213ce87d, queue=awx_private_queue)
2021-09-27 12:14:43,698 DEBUG awx.main.dispatch delivered 01998887-0288-42da-96af-c271213ce87d to worker[182] qsize 0
2021-09-27 12:14:43,701 DEBUG awx.main.dispatch task 01998887-0288-42da-96af-c271213ce87d starting awx.main.tasks.awx_periodic_scheduler(*[])
2021-09-27 12:14:45,745 ERROR awx.main.dispatch Worker failed to run task awx.main.tasks.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 482, in awx_periodic_scheduler
with advisory_lock('awx_periodic_scheduler_lock', wait=False) as acquired:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/pglock.py", line 13, in advisory_lock
with django_pglocks_advisory_lock(*args, **kwargs) as internal_lock:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django_pglocks/__init__.py", line 64, in advisory_lock
cursor = connections[using].cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "awx-postgresql" (10.96.172.217) and accepting
TCP/IP connections on port 5432?
2021-09-27 12:14:45,748 DEBUG awx.main.dispatch task 01998887-0288-42da-96af-c271213ce87d is finished
2021-09-27 12:14:53,659 DEBUG awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(fdf723ca-4848-46e3-903d-f939bf5aa7a5, queue=awx_private_queue)
[2021-09-27 12:14:53,659: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(fdf723ca-4848-46e3-903d-f939bf5aa7a5, queue=awx_private_queue)
2021-09-27 12:14:54,707 DEBUG awx.main.dispatch delivered fdf723ca-4848-46e3-903d-f939bf5aa7a5 to worker[183] qsize 0
2021-09-27 12:14:54,709 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(be1475d6-60ba-47ad-bb2f-9e4411072bf5, queue=dev-awx-0)
[2021-09-27 12:14:54,709: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(be1475d6-60ba-47ad-bb2f-9e4411072bf5, queue=dev-awx-0)
2021-09-27 12:14:54,730 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(af2a402a-e5f6-4acf-81df-3047b90397c9, queue=dev-awx-0)
[2021-09-27 12:14:54,730: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(af2a402a-e5f6-4acf-81df-3047b90397c9, queue=dev-awx-0)
2021-09-27 12:14:54,712 DEBUG awx.main.dispatch task fdf723ca-4848-46e3-903d-f939bf5aa7a5 starting awx.main.scheduler.tasks.run_task_manager(*[])
2021-09-27 12:14:55,732 DEBUG awx.main.scheduler Running Tower task manager.
2021-09-27 12:14:55,765 ERROR awx.main.dispatch failed to write inbound message
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.6/site