Results

By type

          *** Starting uWSGI 2.0.19.1 (64bit) on [Tue Jul 13 01:03:35 2021] ***
compiled with version: 9.3.0 on 16 March 2021 05:56:01
os: Linux-4.19.0-13-cloud-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28)
nodename: onap-oof-has-api-7fbfc68cfb-r7mwg
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 16
current working directory: /app
writing pidfile to /run/conductor/conductor-uwsgi.pid
detected binary path: /usr/local/bin/uwsgi
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /run/conductor/uwsgi.sock fd 3
uwsgi socket 1 bound to TCP address 0.0.0.0:8080 fd 4
Python version: 3.9.1 (default, Dec 11 2020, 14:29:41)  [GCC 9.3.0]
Python main interpreter initialized at 0x555b1065a840
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 510440 bytes (498 KB) for 6 cores
*** Operational MODE: preforking ***
2021-07-13 01:03:37,590||140202084580680|INFO|app|conductor.api.app: [-] Full WSGI config used: /usr/local/etc/conductor/api_paste.ini
2021-07-13 01:03:37,591||140202084580680|INFO|api|conductor.common.music.api: [-] Initializing Music API
WSGI app 0 (mountpoint='') ready in 10 seconds on interpreter 0x555b1065a840 pid: 1 (default app)
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 141, cores: 1)
spawned uWSGI worker 2 (pid: 142, cores: 1)
spawned uWSGI worker 3 (pid: 143, cores: 1)
spawned uWSGI worker 4 (pid: 144, cores: 1)
spawned uWSGI worker 5 (pid: 145, cores: 1)
spawned uWSGI worker 6 (pid: 146, cores: 1)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 147)
Respawned uWSGI worker 2 (new pid: 148)
Respawned uWSGI worker 3 (new pid: 149)
Respawned uWSGI worker 4 (new pid: 150)
Respawned uWSGI worker 5 (new pid: 151)
Respawned uWSGI worker 6 (new pid: 152)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 153)
Respawned uWSGI worker 2 (new pid: 154)
Respawned uWSGI worker 3 (new pid: 155)
Respawned uWSGI worker 4 (new pid: 156)
Respawned uWSGI worker 5 (new pid: 157)
Respawned uWSGI worker 6 (new pid: 158)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 159)
Respawned uWSGI worker 2 (new pid: 160)
Respawned uWSGI worker 3 (new pid: 161)
Respawned uWSGI worker 4 (new pid: 162)
Respawned uWSGI worker 5 (new pid: 163)
Respawned uWSGI worker 6 (new pid: 164)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 165)
Respawned uWSGI worker 2 (new pid: 166)
Respawned uWSGI worker 3 (new pid: 167)
Respawned uWSGI worker 4 (new pid: 168)
Respawned uWSGI worker 5 (new pid: 169)
Respawned uWSGI worker 6 (new pid: 170)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 171)
Respawned uWSGI worker 2 (new pid: 172)
Respawned uWSGI worker 3 (new pid: 173)
Respawned uWSGI worker 4 (new pid: 174)
Respawned uWSGI worker 5 (new pid: 175)
Respawned uWSGI worker 6 (new pid: 176)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 177)
Respawned uWSGI worker 2 (new pid: 178)
Respawned uWSGI worker 3 (new pid: 179)
Respawned uWSGI worker 4 (new pid: 180)
Respawned uWSGI worker 5 (new pid: 181)
Respawned uWSGI worker 6 (new pid: 182)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 183)
Respawned uWSGI worker 2 (new pid: 184)
Respawned uWSGI worker 3 (new pid: 185)
Respawned uWSGI worker 4 (new pid: 186)
Respawned uWSGI worker 5 (new pid: 187)
Respawned uWSGI worker 6 (new pid: 188)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 189)
Respawned uWSGI worker 2 (new pid: 190)
Respawned uWSGI worker 3 (new pid: 191)
Respawned uWSGI worker 4 (new pid: 192)
Respawned uWSGI worker 5 (new pid: 193)
Respawned uWSGI worker 6 (new pid: 194)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 195)
Respawned uWSGI worker 2 (new pid: 196)
Respawned uWSGI worker 3 (new pid: 197)
Respawned uWSGI worker 4 (new pid: 198)
Respawned uWSGI worker 5 (new pid: 199)
Respawned uWSGI worker 6 (new pid: 200)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 201)
Respawned uWSGI worker 2 (new pid: 202)
Respawned uWSGI worker 3 (new pid: 203)
Respawned uWSGI worker 4 (new pid: 204)
Respawned uWSGI worker 5 (new pid: 205)
Respawned uWSGI worker 6 (new pid: 206)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 207)
Respawned uWSGI worker 2 (new pid: 208)
Respawned uWSGI worker 3 (new pid: 209)
Respawned uWSGI worker 4 (new pid: 210)
Respawned uWSGI worker 5 (new pid: 211)
Respawned uWSGI worker 6 (new pid: 212)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 213)
Respawned uWSGI worker 2 (new pid: 214)
Respawned uWSGI worker 3 (new pid: 215)
Respawned uWSGI worker 4 (new pid: 216)
Respawned uWSGI worker 5 (new pid: 217)
Respawned uWSGI worker 6 (new pid: 218)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 219)
Respawned uWSGI worker 2 (new pid: 220)
Respawned uWSGI worker 3 (new pid: 221)
Respawned uWSGI worker 4 (new pid: 222)
Respawned uWSGI worker 5 (new pid: 223)
Respawned uWSGI worker 6 (new pid: 224)
2021-07-13 02:11:12,335||140202084580680|INFO|component|conductor.common.music.messaging.component: [-] Message c68ca9f2-4536-4e92-8402-b5ad82a74520 on topic controller enqueued.
2021-07-13 02:11:14,665||140202084580680|ERROR|errors|conductor.api.controllers.errors: [-] Plan healthcheck not found
[pid: 221|app: 0|req: 1/1] 10.233.76.76 () {46 vars in 586 bytes} [Tue Jul 13 02:11:11 2021] GET /v1/plans/healthcheck => generated 60 bytes in 2804 msecs (HTTP/1.1 404) 8 headers in 202 bytes (1 switches on core 0)
2021-07-13 02:13:42,906||140202084580680|INFO|component|conductor.common.music.messaging.component: [-] Message 29cc1321-0b56-4a2b-97b3-c349d328e905 on topic controller enqueued.
2021-07-13 02:13:45,759||140202084580680|ERROR|errors|conductor.api.controllers.errors: [-] Plan healthcheck not found
[pid: 222|app: 0|req: 1/2] 10.233.74.213 () {46 vars in 589 bytes} [Tue Jul 13 02:13:42 2021] GET /v1/plans/healthcheck => generated 60 bytes in 3509 msecs (HTTP/1.1 404) 8 headers in 202 bytes (1 switches on core 0)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 229)
Respawned uWSGI worker 2 (new pid: 230)
Respawned uWSGI worker 3 (new pid: 231)
Respawned uWSGI worker 4 (new pid: 232)
Respawned uWSGI worker 5 (new pid: 233)
Respawned uWSGI worker 6 (new pid: 234)
2021-07-13 02:15:27,182||140202084580680|INFO|component|conductor.common.music.messaging.component: [-] Message 1a05e21e-f027-4f8c-822b-91d7553d6553 on topic controller enqueued.
2021-07-13 02:17:35,469||140202084580680|INFO|component|conductor.common.music.messaging.component: [-] Message 333d4890-6c49-4e2c-8dce-a5e1446d672d on topic controller enqueued.
2021-07-13 02:17:36,240||140202084580680|ERROR|component|conductor.common.music.messaging.component: [-] Message 1a05e21e-f027-4f8c-822b-91d7553d6553 on topic controller timed out at 120 seconds
2021-07-13 02:17:36,242||140202084580680|CRITICAL|log|conductor: [-] Unhandled error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 141, in lookup_controller
    obj, remainder = find_object(obj, remainder, notfound_handlers,
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 197, in find_object
    raise PecanNotFound
pecan.routing.PecanNotFound

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send
    status, headers, app_iter = self.call_application(
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send
    status, headers, app_iter = self.call_application(
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/opt/has/conductor/conductor/api/middleware.py", line 130, in __call__
    app_iter = self.app(environ, replacement_start_response)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 852, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 692, in __call__
    controller, args, kwargs = self.find_controller(state)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 870, in find_controller
    controller, args, kw = super(Pecan, self).find_controller(_state)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 460, in find_controller
    controller, remainder = self.route(req, self.root, path)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 293, in route
    node, remainder = lookup_controller(node, path, req)
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 158, in lookup_controller
    result = handle_lookup_traversal(obj, remainder)
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 175, in handle_lookup_traversal
    result = obj(*args)
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 311, in _lookup
    return PlansItemController(uuid4), remainder
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 190, in __init__
    self.plan = self.plans_get(plan_id=self.uuid)
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 95, in plans_get
    return self.plan_getid(plan_id)
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 110, in plan_getid
    result = client.call(ctx, method, args)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 229, in call
    response = rpc.response
AttributeError: 'NoneType' object has no attribute 'response'
[pid: 232|app: 0|req: 2/3] 10.233.74.36 () {46 vars in 586 bytes} [Tue Jul 13 02:15:12 2021] GET /v1/plans/healthcheck => generated 0 bytes in 144223 msecs (HTTP/1.1 500) 0 headers in 0 bytes (1 switches on core 0)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 249)
Respawned uWSGI worker 2 (new pid: 250)
Respawned uWSGI worker 3 (new pid: 251)
Respawned uWSGI worker 4 (new pid: 252)
Respawned uWSGI worker 5 (new pid: 253)
2021-07-13 02:19:44,534||140202084580680|ERROR|component|conductor.common.music.messaging.component: [-] Message 333d4890-6c49-4e2c-8dce-a5e1446d672d on topic controller timed out at 120 seconds
2021-07-13 02:19:44,537||140202084580680|CRITICAL|log|conductor: [-] Unhandled error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 141, in lookup_controller
    obj, remainder = find_object(obj, remainder, notfound_handlers,
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 197, in find_object
    raise PecanNotFound
pecan.routing.PecanNotFound

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send
    status, headers, app_iter = self.call_application(
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send
    status, headers, app_iter = self.call_application(
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/opt/has/conductor/conductor/api/middleware.py", line 130, in __call__
    app_iter = self.app(environ, replacement_start_response)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 852, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 692, in __call__
    controller, args, kwargs = self.find_controller(state)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 870, in find_controller
    controller, args, kw = super(Pecan, self).find_controller(_state)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 460, in find_controller
    controller, remainder = self.route(req, self.root, path)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 293, in route
    node, remainder = lookup_controller(node, path, req)
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 158, in lookup_controller
    result = handle_lookup_traversal(obj, remainder)
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 175, in handle_lookup_traversal
    result = obj(*args)
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 311, in _lookup
    return PlansItemController(uuid4), remainder
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 190, in __init__
    self.plan = self.plans_get(plan_id=self.uuid)
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 95, in plans_get
    return self.plan_getid(plan_id)
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 110, in plan_getid
    result = client.call(ctx, method, args)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 229, in call
    response = rpc.response
AttributeError: 'NoneType' object has no attribute 'response'
[pid: 234|app: 0|req: 1/4] 10.233.74.96 () {46 vars in 586 bytes} [Tue Jul 13 02:17:20 2021] GET /v1/plans/healthcheck => generated 0 bytes in 144231 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
Respawned uWSGI worker 6 (new pid: 256)
2021-07-13 02:20:07,251||140202084580680|INFO|component|conductor.common.music.messaging.component: [-] Message acf54e26-25a3-4eb6-9ba2-54f9debd7fd6 on topic controller enqueued.
2021-07-13 02:20:10,245||140202084580680|ERROR|errors|conductor.api.controllers.errors: [-] Plan healthcheck not found
[pid: 252|app: 0|req: 3/5] 10.233.76.252 () {46 vars in 589 bytes} [Tue Jul 13 02:20:07 2021] GET /v1/plans/healthcheck => generated 60 bytes in 3166 msecs (HTTP/1.1 404) 8 headers in 202 bytes (2 switches on core 0)
2021-07-13 02:23:03,662||140202084580680|INFO|component|conductor.common.music.messaging.component: [-] Message 515c730d-a8a0-40c6-a6ef-5b261c6c58eb on topic controller enqueued.
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
worker 5 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 263)
Respawned uWSGI worker 2 (new pid: 264)
Respawned uWSGI worker 3 (new pid: 265)
Respawned uWSGI worker 4 (new pid: 266)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 6 (new pid: 270)
2021-07-13 02:25:12,755||140202084580680|ERROR|component|conductor.common.music.messaging.component: [-] Message 515c730d-a8a0-40c6-a6ef-5b261c6c58eb on topic controller timed out at 120 seconds
2021-07-13 02:25:12,756||140202084580680|CRITICAL|log|conductor: [-] Unhandled error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 141, in lookup_controller
    obj, remainder = find_object(obj, remainder, notfound_handlers,
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 197, in find_object
    raise PecanNotFound
pecan.routing.PecanNotFound

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send
    status, headers, app_iter = self.call_application(
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/oslo_middleware/base.py", line 124, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1313, in send
    status, headers, app_iter = self.call_application(
  File "/usr/local/lib/python3.9/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/opt/has/conductor/conductor/api/middleware.py", line 130, in __call__
    app_iter = self.app(environ, replacement_start_response)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 852, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 692, in __call__
    controller, args, kwargs = self.find_controller(state)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 870, in find_controller
    controller, args, kw = super(Pecan, self).find_controller(_state)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 460, in find_controller
    controller, remainder = self.route(req, self.root, path)
  File "/usr/local/lib/python3.9/site-packages/pecan/core.py", line 293, in route
    node, remainder = lookup_controller(node, path, req)
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 158, in lookup_controller
    result = handle_lookup_traversal(obj, remainder)
  File "/usr/local/lib/python3.9/site-packages/pecan/routing.py", line 175, in handle_lookup_traversal
    result = obj(*args)
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 311, in _lookup
    return PlansItemController(uuid4), remainder
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 190, in __init__
    self.plan = self.plans_get(plan_id=self.uuid)
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 95, in plans_get
    return self.plan_getid(plan_id)
  File "/opt/has/conductor/conductor/api/controllers/v1/plans.py", line 110, in plan_getid
    result = client.call(ctx, method, args)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 229, in call
    response = rpc.response
AttributeError: 'NoneType' object has no attribute 'response'
[pid: 253|app: 0|req: 1/6] 10.233.76.83 () {46 vars in 586 bytes} [Tue Jul 13 02:22:48 2021] GET /v1/plans/healthcheck => generated 0 bytes in 144262 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
Respawned uWSGI worker 5 (new pid: 272)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 273)
Respawned uWSGI worker 2 (new pid: 274)
Respawned uWSGI worker 3 (new pid: 275)
Respawned uWSGI worker 4 (new pid: 276)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 6 (new pid: 277)
worker 5 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 5 (new pid: 278)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 279)
Respawned uWSGI worker 2 (new pid: 280)
Respawned uWSGI worker 3 (new pid: 281)
Respawned uWSGI worker 4 (new pid: 282)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 6 (new pid: 283)
worker 5 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 5 (new pid: 284)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 285)
Respawned uWSGI worker 2 (new pid: 286)
Respawned uWSGI worker 3 (new pid: 287)
Respawned uWSGI worker 4 (new pid: 288)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 6 (new pid: 289)
worker 5 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 5 (new pid: 290)
worker 1 lifetime reached, it was running for 301 second(s)
worker 2 lifetime reached, it was running for 301 second(s)
worker 3 lifetime reached, it was running for 301 second(s)
worker 4 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 1 (new pid: 291)
Respawned uWSGI worker 2 (new pid: 292)
Respawned uWSGI worker 3 (new pid: 293)
Respawned uWSGI worker 4 (new pid: 294)
worker 6 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 6 (new pid: 295)
worker 5 lifetime reached, it was running for 301 second(s)
Respawned uWSGI worker 5 (new pid: 296)