*** Starting uWSGI 2.0.19.1 (64bit) on [Thu May 20 07:06:52 2021] *** compiled with version: 9.3.0 on 16 March 2021 05:56:01 os: Linux-4.19.0-13-cloud-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) nodename: onap-oof-has-api-7fbfc68cfb-kwllq machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 16 current working directory: /app writing pidfile to /run/conductor/conductor-uwsgi.pid detected binary path: /usr/local/bin/uwsgi your memory page size is 4096 bytes detected max file descriptor number: 1048576 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to UNIX address /run/conductor/uwsgi.sock fd 3 uwsgi socket 1 bound to TCP address 0.0.0.0:8080 fd 4 Python version: 3.9.1 (default, Dec 11 2020, 14:29:41) [GCC 9.3.0] Python main interpreter initialized at 0x5555f96a2f60 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 510440 bytes (498 KB) for 6 cores *** Operational MODE: preforking *** 2021-05-20 07:06:55,082||139960009055560|INFO|app|conductor.api.app: [-] Full WSGI config used: /usr/local/etc/conductor/api_paste.ini 2021-05-20 07:06:55,085||139960009055560|INFO|api|conductor.common.music.api: [-] Initializing Music API WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x5555f96a2f60 pid: 1 (default app) spawned uWSGI master process (pid: 1) spawned uWSGI worker 1 (pid: 141, cores: 1) spawned uWSGI worker 2 (pid: 142, cores: 1) spawned uWSGI worker 3 (pid: 143, cores: 1) spawned uWSGI worker 4 (pid: 144, cores: 1) spawned uWSGI worker 5 (pid: 145, cores: 1) spawned uWSGI worker 6 (pid: 146, cores: 1) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 147) Respawned uWSGI worker 2 (new pid: 148) Respawned uWSGI worker 3 (new pid: 149) Respawned uWSGI worker 4 (new pid: 150) Respawned uWSGI worker 5 (new pid: 151) Respawned uWSGI worker 6 (new pid: 152) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 153) Respawned uWSGI worker 2 (new pid: 154) Respawned uWSGI worker 3 (new pid: 155) Respawned uWSGI worker 4 (new pid: 156) Respawned uWSGI worker 5 (new pid: 157) Respawned uWSGI worker 6 (new pid: 158) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 159) Respawned uWSGI worker 2 (new pid: 160) Respawned uWSGI worker 3 (new pid: 161) Respawned uWSGI worker 4 (new pid: 162) Respawned uWSGI worker 5 (new pid: 163) Respawned uWSGI worker 6 (new pid: 164) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 165) Respawned uWSGI worker 2 (new pid: 166) Respawned uWSGI worker 3 (new pid: 167) Respawned uWSGI worker 4 (new pid: 168) Respawned uWSGI worker 5 (new pid: 169) Respawned uWSGI worker 6 (new pid: 170) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 171) Respawned uWSGI worker 2 (new pid: 172) Respawned uWSGI worker 3 (new pid: 173) Respawned uWSGI worker 4 (new pid: 174) Respawned uWSGI worker 5 (new pid: 175) Respawned uWSGI worker 6 (new pid: 176) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 177) Respawned uWSGI worker 2 (new pid: 178) Respawned uWSGI worker 3 (new pid: 179) Respawned uWSGI worker 4 (new pid: 180) Respawned uWSGI worker 5 (new pid: 181) Respawned uWSGI worker 6 (new pid: 182) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 183) Respawned uWSGI worker 2 (new pid: 184) Respawned uWSGI worker 3 (new pid: 185) Respawned uWSGI worker 4 (new pid: 186) Respawned uWSGI worker 5 (new pid: 187) Respawned uWSGI worker 6 (new pid: 188) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 189) Respawned uWSGI worker 2 (new pid: 190) Respawned uWSGI worker 3 (new pid: 191) Respawned uWSGI worker 4 (new pid: 192) Respawned uWSGI worker 5 (new pid: 193) Respawned uWSGI worker 6 (new pid: 194) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 195) Respawned uWSGI worker 2 (new pid: 196) Respawned uWSGI worker 3 (new pid: 197) Respawned uWSGI worker 4 (new pid: 198) Respawned uWSGI worker 5 (new pid: 199) Respawned uWSGI worker 6 (new pid: 200) 2021-05-20 07:53:30,061||139960009055560|INFO|component|conductor.common.music.messaging.component: [-] Message f163141a-ef11-4942-b2fd-06e1a2b33823 on topic controller enqueued. [pid: 200|app: 0|req: 1/1] 10.233.73.146 () {46 vars in 589 bytes} [Thu May 20 07:53:29 2021] GET /v1/plans/healthcheck => generated 159 bytes in 2779 msecs (HTTP/1.1 200) 5 headers in 134 bytes (2 switches on core 0) 2021-05-20 07:56:03,506||139960009055560|INFO|component|conductor.common.music.messaging.component: [-] Message 83f5148d-d8e6-4502-aee5-1928fcb1ce52 on topic controller enqueued. worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 235) Respawned uWSGI worker 2 (new pid: 236) Respawned uWSGI worker 3 (new pid: 237) Respawned uWSGI worker 4 (new pid: 238) Respawned uWSGI worker 5 (new pid: 239) 2021-05-20 07:57:52,481||139960009055560|INFO|component|conductor.common.music.messaging.component: [-] Message 4cd814a6-4722-42db-a9b3-24504255d716 on topic controller enqueued. [pid: 200|app: 0|req: 2/2] 10.233.78.241 () {46 vars in 589 bytes} [Thu May 20 07:56:00 2021] GET /v1/plans/healthcheck => generated 159 bytes in 125334 msecs (HTTP/1.1 200) 5 headers in 134 bytes (2 switches on core 0) [pid: 238|app: 0|req: 1/3] 10.233.68.70 () {46 vars in 586 bytes} [Thu May 20 07:57:51 2021] GET /v1/plans/healthcheck => generated 159 bytes in 14858 msecs (HTTP/1.1 200) 5 headers in 134 bytes (2 switches on core 0) Respawned uWSGI worker 6 (new pid: 294) 2021-05-20 07:59:42,335||139960009055560|INFO|component|conductor.common.music.messaging.component: [-] Message 9cee60ec-01e7-46f0-8ef7-c02c6c5daef8 on topic controller enqueued. [pid: 237|app: 0|req: 1/4] 10.233.78.158 () {46 vars in 589 bytes} [Thu May 20 07:59:38 2021] GET /v1/plans/healthcheck => generated 159 bytes in 10469 msecs (HTTP/1.1 200) 5 headers in 134 bytes (1 switches on core 0) 2021-05-20 08:01:33,896||139960009055560|INFO|component|conductor.common.music.messaging.component: [-] Message ba9cf1f5-df3f-44b8-9aa0-6d450e7fbd54 on topic controller enqueued. [pid: 294|app: 0|req: 3/5] 10.233.68.252 () {46 vars in 589 bytes} [Thu May 20 08:01:33 2021] GET /v1/plans/healthcheck => generated 159 bytes in 20875 msecs (HTTP/1.1 200) 5 headers in 134 bytes (2 switches on core 0) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 319) Respawned uWSGI worker 2 (new pid: 320) Respawned uWSGI worker 3 (new pid: 321) Respawned uWSGI worker 4 (new pid: 322) Respawned uWSGI worker 5 (new pid: 323) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 6 (new pid: 324) 2021-05-20 08:04:21,205||139960009055560|INFO|component|conductor.common.music.messaging.component: [-] Message 1eb07ecc-1232-43aa-b6cc-dd62c500985c on topic controller enqueued. 2021-05-20 08:06:24,265||139960009055560|ERROR|component|conductor.common.music.messaging.component: [-] Message 1eb07ecc-1232-43aa-b6cc-dd62c500985c on topic controller timed out at 120 seconds 2021-05-20 08:06:24,266||139960009055560|CRITICAL|log|conductor: [-] Unhandled error Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 141, in lookup_controller obj, remainder = find_object(obj, remainder, notfound_handlers, File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 197, in find_object raise PecanNotFound pecan.routing.PecanNotFound During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__ resp = self.call_func(req, *args, **kw) File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func return self.func(req, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__ response = req.get_response(self.application) File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send status, headers, app_iter = self.call_application( File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application app_iter = application(self.environ, start_response) File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__ resp = self.call_func(req, *args, **kw) File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func return self.func(req, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__ response = req.get_response(self.application) File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send status, headers, app_iter = self.call_application( File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application app_iter = application(self.environ, start_response) File "/usr/local/lib/python3.9/site-packages/pecan/middleware/recursive.py", line 56, in __call__ return self.application(environ, start_response) File "/opt/has/conductor/conductor/api/middleware.py", line 130, in __call__ app_iter = self.app(environ, replacement_start_response) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 852, in __call__ return super(Pecan, self).__call__(environ, start_response) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 692, in __call__ controller, args, kwargs = self.find_controller(state) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 870, in find_controller controller, args, kw = super(Pecan, self).find_controller(_state) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 460, in find_controller controller, remainder = self.route(req, self.root, path) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 293, in route node, remainder = lookup_controller(node, path, req) File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 158, in lookup_controller result = handle_lookup_traversal(obj, remainder) File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 175, in handle_lookup_traversal result = obj(*args) File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 311, in _lookup return PlansItemController(uuid4), remainder File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 190, in __init__ self.plan = self.plans_get(plan_id=self.uuid) File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 95, in plans_get return self.plan_getid(plan_id) File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 110, in plan_getid result = client.call(ctx, method, args) File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 229, in call response = rpc.response AttributeError: 'NoneType' object has no attribute 'response' [pid: 320|app: 0|req: 1/6] 10.233.78.81 () {46 vars in 586 bytes} [Thu May 20 08:04:16 2021] GET /v1/plans/healthcheck => generated 0 bytes in 127636 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 3 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 405) Respawned uWSGI worker 2 (new pid: 406) Respawned uWSGI worker 4 (new pid: 407) Respawned uWSGI worker 5 (new pid: 408) 2021-05-20 08:07:08,907||139960009055560|INFO|component|conductor.common.music.messaging.component: [-] Message 02794bac-6c44-4587-bef3-47d755e798cc on topic controller enqueued. worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 6 (new pid: 460) 2021-05-20 08:09:08,983||139960009055560|ERROR|component|conductor.common.music.messaging.component: [-] Message 02794bac-6c44-4587-bef3-47d755e798cc on topic controller timed out at 120 seconds 2021-05-20 08:09:08,984||139960009055560|CRITICAL|log|conductor: [-] Unhandled error Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 141, in lookup_controller obj, remainder = find_object(obj, remainder, notfound_handlers, File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 197, in find_object raise PecanNotFound pecan.routing.PecanNotFound During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__ resp = self.call_func(req, *args, **kw) File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func return self.func(req, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__ response = req.get_response(self.application) File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send status, headers, app_iter = self.call_application( File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application app_iter = application(self.environ, start_response) File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__ resp = self.call_func(req, *args, **kw) File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func return self.func(req, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__ response = req.get_response(self.application) File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send status, headers, app_iter = self.call_application( File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application app_iter = application(self.environ, start_response) File "/usr/local/lib/python3.9/site-packages/pecan/middleware/recursive.py", line 56, in __call__ return self.application(environ, start_response) File "/opt/has/conductor/conductor/api/middleware.py", line 130, in __call__ app_iter = self.app(environ, replacement_start_response) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 852, in __call__ return super(Pecan, self).__call__(environ, start_response) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 692, in __call__ controller, args, kwargs = self.find_controller(state) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 870, in find_controller controller, args, kw = super(Pecan, self).find_controller(_state) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 460, in find_controller controller, remainder = self.route(req, self.root, path) File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 293, in route node, remainder = lookup_controller(node, path, req) File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 158, in lookup_controller result = handle_lookup_traversal(obj, remainder) File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 175, in handle_lookup_traversal result = obj(*args) File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 311, in _lookup return PlansItemController(uuid4), remainder File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 190, in __init__ self.plan = self.plans_get(plan_id=self.uuid) File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 95, in plans_get return self.plan_getid(plan_id) File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 110, in plan_getid result = client.call(ctx, method, args) File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 229, in call response = rpc.response AttributeError: 'NoneType' object has no attribute 'response' [pid: 321|app: 0|req: 2/7] 10.233.68.118 () {46 vars in 589 bytes} [Thu May 20 08:06:56 2021] GET /v1/plans/healthcheck => generated 0 bytes in 132253 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0) Respawned uWSGI worker 3 (new pid: 490) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 491) Respawned uWSGI worker 2 (new pid: 492) Respawned uWSGI worker 4 (new pid: 493) Respawned uWSGI worker 5 (new pid: 494) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 6 (new pid: 495) worker 3 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 3 (new pid: 496) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 497) Respawned uWSGI worker 2 (new pid: 498) Respawned uWSGI worker 4 (new pid: 499) Respawned uWSGI worker 5 (new pid: 500) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 6 (new pid: 501) worker 3 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 3 (new pid: 502) worker 1 lifetime reached, it was running for 301 second(s) worker 2 lifetime reached, it was running for 301 second(s) worker 4 lifetime reached, it was running for 301 second(s) worker 5 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 1 (new pid: 503) Respawned uWSGI worker 2 (new pid: 504) Respawned uWSGI worker 4 (new pid: 505) Respawned uWSGI worker 5 (new pid: 506) worker 6 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 6 (new pid: 507) worker 3 lifetime reached, it was running for 301 second(s) Respawned uWSGI worker 3 (new pid: 508)