2022/03/15 10:05:06 Etcd Client Initialization failed with error: Error creating etcd client: dial tcp: lookup multicloud-k8s-etcd on 169.254.25.10:53: no such host
2022/03/15 10:05:06 Starting Kubernetes Multicloud API
2022/03/15 10:05:06 Error Getting TLS Configuration. Starting without TLS...
10.233.68.107 - - [15/Mar/2022:11:10:18 +0000] "GET /v1/healthcheck HTTP/1.1" 200 0
127.0.0.1 - - [15/Mar/2022:11:13:11 +0000] "POST /v1/rb/definition HTTP/1.1" 201 278
127.0.0.1 - - [15/Mar/2022:11:14:44 +0000] "POST /v1/rb/definition HTTP/1.1" 201 270
127.0.0.1 - - [15/Mar/2022:11:17:12 +0000] "POST /v1/rb/definition HTTP/1.1" 201 270
10.233.68.107 - - [15/Mar/2022:11:27:35 +0000] "GET /v1/connectivity-info/k8sregion HTTP/1.1" 500 74
10.233.68.107 - - [15/Mar/2022:11:27:35 +0000] "POST /v1/connectivity-info HTTP/1.1" 201 7669
10.233.68.107 - - [15/Mar/2022:11:27:35 +0000] "GET /v1/connectivity-info/k8sregion HTTP/1.1" 200 7669
10.233.68.107 - - [15/Mar/2022:11:27:35 +0000] "POST /v1/rb/definition HTTP/1.1" 201 125
10.233.68.107 - - [15/Mar/2022:11:27:35 +0000] "GET /v1/rb/definition/pnf-cnf-rb-8165bc9e-ad39-4550-8bdd-8850ffe1db74/v1 HTTP/1.1" 200 125
10.233.68.107 - - [15/Mar/2022:11:27:35 +0000] "POST /v1/rb/definition/pnf-cnf-rb-8165bc9e-ad39-4550-8bdd-8850ffe1db74/v1/content HTTP/1.1" 200 0
10.233.68.107 - - [15/Mar/2022:11:27:35 +0000] "GET /v1/rb/definition/pnf-cnf-rb-8165bc9e-ad39-4550-8bdd-8850ffe1db74/v1 HTTP/1.1" 200 138
10.233.68.107 - - [15/Mar/2022:11:27:36 +0000] "POST /v1/rb/definition/pnf-cnf-rb-8165bc9e-ad39-4550-8bdd-8850ffe1db74/v1/profile HTTP/1.1" 201 314
10.233.68.107 - - [15/Mar/2022:11:27:36 +0000] "GET /v1/rb/definition/pnf-cnf-rb-8165bc9e-ad39-4550-8bdd-8850ffe1db74/v1/profile/pnf-cnf-profile-0a660328-7877-4d62-8f27-a7e1fae91724 HTTP/1.1" 200 314
10.233.68.107 - - [15/Mar/2022:11:27:36 +0000] "POST /v1/rb/definition/pnf-cnf-rb-8165bc9e-ad39-4550-8bdd-8850ffe1db74/v1/profile/pnf-cnf-profile-0a660328-7877-4d62-8f27-a7e1fae91724/content HTTP/1.1" 200 0
2022/03/15 11:27:36 Error reading name cache from Database:  Error finding master table: mongo: no documents in result
2022/03/15 11:27:36 Error Reading from DB:  Reading cache from DB: Error finding master table: mongo: no documents in result
2022/03/15 11:27:36 Main rss info
2022/03/15 11:27:36   Path: /tmp/helm-tmpl-094790399/manifest-0
2022/03/15 11:27:36     Kind: Service
2022/03/15 11:27:36   Path: /tmp/helm-tmpl-094790399/manifest-1
2022/03/15 11:27:36     Kind: Service
2022/03/15 11:27:36   Path: /tmp/helm-tmpl-094790399/manifest-2
2022/03/15 11:27:36     Kind: Pod
2022/03/15 11:27:36   Path: /tmp/helm-tmpl-094790399/manifest-3
2022/03/15 11:27:36     Kind: Deployment
2022/03/15 11:27:36 Crd rss info
2022/03/15 11:27:36 Hook info
{"level":"info","msg":"Creating Namespace","namespace":"onap-tests","time":"2022-03-15T11:27:36Z"}
2022/03/15 11:27:36 Namespace (onap-tests) created
{"filepath":"/tmp/helm-tmpl-094790399/manifest-0","level":"info","msg":"Processing Kubernetes Resource","time":"2022-03-15T11:27:36Z"}
{"gvk":{"Group":"","Version":"v1","Kind":"Service"},"level":"info","msg":"Created Kubernetes Resource","resource":"pnf-macro-test-simulator-mongo","time":"2022-03-15T11:27:36Z"}
{"filepath":"/tmp/helm-tmpl-094790399/manifest-1","level":"info","msg":"Processing Kubernetes Resource","time":"2022-03-15T11:27:36Z"}
{"gvk":{"Group":"","Version":"v1","Kind":"Service"},"level":"info","msg":"Created Kubernetes Resource","resource":"pnf-macro-test-simulator","time":"2022-03-15T11:27:36Z"}
{"filepath":"/tmp/helm-tmpl-094790399/manifest-2","level":"info","msg":"Processing Kubernetes Resource","time":"2022-03-15T11:27:36Z"}
2022/03/15 11:27:36 No plugin for kind Pod found. Using generic Plugin
2022/03/15 11:27:36 Error converting template to map
{"gvk":{"Group":"","Version":"v1","Kind":"Pod"},"level":"info","msg":"Created Kubernetes Resource","resource":"pnf-macro-test-simulator","time":"2022-03-15T11:27:36Z"}
{"filepath":"/tmp/helm-tmpl-094790399/manifest-3","level":"info","msg":"Processing Kubernetes Resource","time":"2022-03-15T11:27:36Z"}
2022/03/15 11:27:36 No plugin for kind Deployment found. Using generic Plugin
{"gvk":{"Group":"apps","Version":"v1","Kind":"Deployment"},"level":"info","msg":"Created Kubernetes Resource","resource":"pnf-macro-test-simulator-mongo","time":"2022-03-15T11:27:36Z"}
10.233.68.107 - - [15/Mar/2022:11:27:36 +0000] "POST /v1/instance HTTP/1.1" 201 738
10.233.68.107 - - [15/Mar/2022:11:28:41 +0000] "GET /v1/connectivity-info/k8sregion HTTP/1.1" 200 7669
10.233.68.107 - - [15/Mar/2022:11:28:41 +0000] "POST /v1/rb/definition HTTP/1.1" 201 124
10.233.68.107 - - [15/Mar/2022:11:28:41 +0000] "GET /v1/rb/definition/cds-ms-rb-216346a5-ba79-404c-9b6d-cea5f3ca5b72/v1 HTTP/1.1" 200 124
10.233.68.107 - - [15/Mar/2022:11:28:41 +0000] "POST /v1/rb/definition/cds-ms-rb-216346a5-ba79-404c-9b6d-cea5f3ca5b72/v1/content HTTP/1.1" 200 0
10.233.68.107 - - [15/Mar/2022:11:28:41 +0000] "GET /v1/rb/definition/cds-ms-rb-216346a5-ba79-404c-9b6d-cea5f3ca5b72/v1 HTTP/1.1" 200 138
10.233.68.107 - - [15/Mar/2022:11:28:41 +0000] "POST /v1/rb/definition/cds-ms-rb-216346a5-ba79-404c-9b6d-cea5f3ca5b72/v1/profile HTTP/1.1" 201 305
10.233.68.107 - - [15/Mar/2022:11:28:41 +0000] "GET /v1/rb/definition/cds-ms-rb-216346a5-ba79-404c-9b6d-cea5f3ca5b72/v1/profile/cds-ms-prof-43718cef-fa90-4918-9ab1-7193c96abc9c HTTP/1.1" 200 305
10.233.68.107 - - [15/Mar/2022:11:28:41 +0000] "POST /v1/rb/definition/cds-ms-rb-216346a5-ba79-404c-9b6d-cea5f3ca5b72/v1/profile/cds-ms-prof-43718cef-fa90-4918-9ab1-7193c96abc9c/content HTTP/1.1" 200 0
2022/03/15 11:28:41 Main rss info
2022/03/15 11:28:41   Path: /tmp/helm-tmpl-902985419/manifest-0
2022/03/15 11:28:41     Kind: Service
2022/03/15 11:28:41   Path: /tmp/helm-tmpl-902985419/manifest-1
2022/03/15 11:28:41     Kind: Deployment
2022/03/15 11:28:41 Crd rss info
2022/03/15 11:28:41 Hook info
{"filepath":"/tmp/helm-tmpl-902985419/manifest-0","level":"info","msg":"Processing Kubernetes Resource","time":"2022-03-15T11:28:41Z"}
{"gvk":{"Group":"","Version":"v1","Kind":"Service"},"level":"info","msg":"Created Kubernetes Resource","resource":"cds-regression-mockserver","time":"2022-03-15T11:28:41Z"}
{"filepath":"/tmp/helm-tmpl-902985419/manifest-1","level":"info","msg":"Processing Kubernetes Resource","time":"2022-03-15T11:28:41Z"}
2022/03/15 11:28:41 No plugin for kind Deployment found. Using generic Plugin
{"gvk":{"Group":"apps","Version":"v1","Kind":"Deployment"},"level":"info","msg":"Created Kubernetes Resource","resource":"cds-cds-mockserver","time":"2022-03-15T11:28:41Z"}
10.233.68.107 - - [15/Mar/2022:11:28:41 +0000] "POST /v1/instance HTTP/1.1" 201 542
2022/03/15 11:29:59 [Config Cleanup] Instance festive_cartwright
2022/03/15 11:29:59 [scheduleResources]: START thread
{"error":"Cleanup Config Resources: Retrieving active config list info: Get Config DB Entry: Get Config DB List: Etcd Client not initialized","level":"error","msg":"Error Deleting Instance","time":"2022-03-15T11:30:04Z"}
10.233.68.107 - - [15/Mar/2022:11:29:59 +0000] "DELETE /v1/instance/festive_cartwright HTTP/1.1" 500 131
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "GET /v1/connectivity-info/k8sregion HTTP/1.1" 200 7669
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "POST /v1/rb/definition HTTP/1.1" 201 124
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "GET /v1/rb/definition/cds-ms-rb-f41374a3-d6c0-4612-8ad1-89782a2b8e73/v1 HTTP/1.1" 200 124
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "POST /v1/rb/definition/cds-ms-rb-f41374a3-d6c0-4612-8ad1-89782a2b8e73/v1/content HTTP/1.1" 200 0
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "GET /v1/rb/definition/cds-ms-rb-f41374a3-d6c0-4612-8ad1-89782a2b8e73/v1 HTTP/1.1" 200 138
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "POST /v1/rb/definition/cds-ms-rb-f41374a3-d6c0-4612-8ad1-89782a2b8e73/v1/profile HTTP/1.1" 201 305
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "GET /v1/rb/definition/cds-ms-rb-f41374a3-d6c0-4612-8ad1-89782a2b8e73/v1/profile/cds-ms-prof-d98c1e31-5cea-4703-b2b3-193d00c3d25b HTTP/1.1" 200 305
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "POST /v1/rb/definition/cds-ms-rb-f41374a3-d6c0-4612-8ad1-89782a2b8e73/v1/profile/cds-ms-prof-d98c1e31-5cea-4703-b2b3-193d00c3d25b/content HTTP/1.1" 200 0
2022/03/15 11:32:52 Main rss info
2022/03/15 11:32:52   Path: /tmp/helm-tmpl-982244665/manifest-0
2022/03/15 11:32:52     Kind: Service
2022/03/15 11:32:52   Path: /tmp/helm-tmpl-982244665/manifest-1
2022/03/15 11:32:52     Kind: Deployment
2022/03/15 11:32:52 Crd rss info
2022/03/15 11:32:52 Hook info
{"filepath":"/tmp/helm-tmpl-982244665/manifest-0","level":"info","msg":"Processing Kubernetes Resource","time":"2022-03-15T11:32:52Z"}
{"error":"Create Service error: Service \"cds-regression-mockserver\" is invalid: spec.ports[0].nodePort: Invalid value: 30726: provided port is already allocated","filepath":"/tmp/helm-tmpl-982244665/manifest-0","gvk":{"Group":"","Version":"v1","Kind":"Service"},"level":"error","msg":"Error Creating Resource","time":"2022-03-15T11:32:52Z"}
2022/03/15 11:32:52   Instance: clever_benz, Main rss are failed, skip post-install and remove instance in DB
2022/03/15 11:32:52 Deleted Obj with ID ObjectID("62307964777bf75510302cd4")
2022/03/15 11:32:52 Deleted Obj with ID ObjectID("62307964fa373c1de9fe53e0")
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "POST /v1/instance HTTP/1.1" 500 252
{"error":"Create Kubernetes Resources: Error creating kind: /v1, Kind=Service: Error in plugin Service plugin: Create Service error: Service \"cds-regression-mockserver\" is invalid: spec.ports[0].nodePort: Invalid value: 30726: provided port is already allocated","level":"error","msg":"Error Creating Resource","resource":{"rb-name":"cds-ms-rb-f41374a3-d6c0-4612-8ad1-89782a2b8e73","rb-version":"v1","profile-name":"cds-ms-prof-d98c1e31-5cea-4703-b2b3-193d00c3d25b","release-name":"","cloud-region":"k8sregion","labels":{},"override-values":{}},"time":"2022-03-15T11:32:52Z"}
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "GET /v1/connectivity-info/k8sregion HTTP/1.1" 200 7669
2022/03/15 11:32:52 Deleted Obj with ID ObjectID("62307827777bf75510302cbb")
2022/03/15 11:32:52 Deleted Obj with ID ObjectID("62307828fa373c1de9fe5251")
10.233.68.107 - - [15/Mar/2022:11:32:52 +0000] "DELETE /v1/connectivity-info/k8sregion HTTP/1.1" 204 0
{"error":"Getting CloudRegion Information: Get kubeconfig file: Downloading kubeconfig: Getting Connection info: Get Connection: Error finding master table: mongo: no documents in result","level":"error","msg":"Error Deleting Instance","time":"2022-03-15T11:35:28Z"}
10.233.68.107 - - [15/Mar/2022:11:35:28 +0000] "DELETE /v1/instance/infallible_wescoff HTTP/1.1" 500 177
10.233.68.107 - - [15/Mar/2022:11:39:05 +0000] "GET /v1/connectivity-info/k8sregion HTTP/1.1" 500 74
10.233.68.107 - - [15/Mar/2022:11:39:05 +0000] "POST /v1/connectivity-info HTTP/1.1" 201 7669
10.233.68.107 - - [15/Mar/2022:11:39:05 +0000] "GET /v1/connectivity-info/k8sregion HTTP/1.1" 200 7669
10.233.68.107 - - [15/Mar/2022:11:39:05 +0000] "POST /v1/rb/definition HTTP/1.1" 201 125
10.233.68.107 - - [15/Mar/2022:11:39:05 +0000] "GET /v1/rb/definition/pnf-cnf-rb-eb38bbfd-5e2d-486c-bfb6-e55c9140dd48/v1 HTTP/1.1" 200 125
10.233.68.107 - - [15/Mar/2022:11:39:05 +0000] "POST /v1/rb/definition/pnf-cnf-rb-eb38bbfd-5e2d-486c-bfb6-e55c9140dd48/v1/content HTTP/1.1" 200 0
10.233.68.107 - - [15/Mar/2022:11:39:05 +0000] "GET /v1/rb/definition/pnf-cnf-rb-eb38bbfd-5e2d-486c-bfb6-e55c9140dd48/v1 HTTP/1.1" 200 138
10.233.68.107 - - [15/Mar/2022:11:39:05 +0000] "POST /v1/rb/definition/pnf-cnf-rb-eb38bbfd-5e2d-486c-bfb6-e55c9140dd48/v1/profile HTTP/1.1" 201 314
10.233.68.107 - - [15/Mar/2022:11:39:06 +0000] "GET /v1/rb/definition/pnf-cnf-rb-eb38bbfd-5e2d-486c-bfb6-e55c9140dd48/v1/profile/pnf-cnf-profile-235513b8-51cd-4219-a540-3e74d6c4f9b6 HTTP/1.1" 200 314
10.233.68.107 - - [15/Mar/2022:11:39:06 +0000] "POST /v1/rb/definition/pnf-cnf-rb-eb38bbfd-5e2d-486c-bfb6-e55c9140dd48/v1/profile/pnf-cnf-profile-235513b8-51cd-4219-a540-3e74d6c4f9b6/content HTTP/1.1" 200 0
2022/03/15 11:39:06 Main rss info
2022/03/15 11:39:06   Path: /tmp/helm-tmpl-057145016/manifest-0
2022/03/15 11:39:06     Kind: Service
2022/03/15 11:39:06   Path: /tmp/helm-tmpl-057145016/manifest-1
2022/03/15 11:39:06     Kind: Service
2022/03/15 11:39:06   Path: /tmp/helm-tmpl-057145016/manifest-2
2022/03/15 11:39:06     Kind: Pod
2022/03/15 11:39:06   Path: /tmp/helm-tmpl-057145016/manifest-3
2022/03/15 11:39:06     Kind: Deployment
2022/03/15 11:39:06 Crd rss info
2022/03/15 11:39:06 Hook info
{"filepath":"/tmp/helm-tmpl-057145016/manifest-0","level":"info","msg":"Processing Kubernetes Resource","time":"2022-03-15T11:39:06Z"}
{"error":"Create Service error: services \"pnf-macro-test-simulator-mongo\" already exists","filepath":"/tmp/helm-tmpl-057145016/manifest-0","gvk":{"Group":"","Version":"v1","Kind":"Service"},"level":"error","msg":"Error Creating Resource","time":"2022-03-15T11:39:06Z"}
2022/03/15 11:39:06   Instance: blissful_swirles, Main rss are failed, skip post-install and remove instance in DB
2022/03/15 11:39:06 Deleted Obj with ID ObjectID("62307ada777bf75510302cdd")
2022/03/15 11:39:06 Deleted Obj with ID ObjectID("62307adafa373c1de9fe5512")
{"error":"Create Kubernetes Resources: Error creating kind: /v1, Kind=Service: Error in plugin Service plugin: Create Service error: services \"pnf-macro-test-simulator-mongo\" already exists","level":"error","msg":"Error Creating Resource","resource":{"rb-name":"pnf-cnf-rb-eb38bbfd-5e2d-486c-bfb6-e55c9140dd48","rb-version":"v1","profile-name":"pnf-cnf-profile-235513b8-51cd-4219-a540-3e74d6c4f9b6","release-name":"","cloud-region":"k8sregion","labels":{},"override-values":{}},"time":"2022-03-15T11:39:06Z"}
10.233.68.107 - - [15/Mar/2022:11:39:06 +0000] "POST /v1/instance HTTP/1.1" 500 180
10.233.68.107 - - [15/Mar/2022:11:39:06 +0000] "GET /v1/connectivity-info/k8sregion HTTP/1.1" 200 7669
2022/03/15 11:39:06 Deleted Obj with ID ObjectID("62307ad9777bf75510302cd5")
2022/03/15 11:39:06 Deleted Obj with ID ObjectID("62307adafa373c1de9fe54d7")
10.233.68.107 - - [15/Mar/2022:11:39:06 +0000] "DELETE /v1/connectivity-info/k8sregion HTTP/1.1" 204 0