Started by timer
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on prd-centos8-builder-2c-1g-76326 (centos8-builder-2c-1g) in workspace /w/workspace/ci-management-openstack-cron
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-7ykuWy6I4QMj/agent.21130
SSH_AGENT_PID=21131
[ssh-agent] Started.
Running ssh-add (command line suppressed)
Identity added: /w/workspace/ci-management-openstack-cron@tmp/private_key_3949797303349119274.key (/w/workspace/ci-management-openstack-cron@tmp/private_key_3949797303349119274.key)
[ssh-agent] Using credentials onap-jobbuiler (Gerrit user)
The recommended git tool is: NONE
using credential onap-jenkins-ssh
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository git://cloud.onap.org/mirror/ci-management
> git init /w/workspace/ci-management-openstack-cron # timeout=10
Fetching upstream changes from git://cloud.onap.org/mirror/ci-management
> git --version # timeout=10
> git --version # 'git version 2.39.0'
using GIT_SSH to set credentials Gerrit user
[INFO] Currently running in a labeled security context
[INFO] Currently SELinux is 'enforcing' on the host
> /usr/bin/chcon --type=ssh_home_t /w/workspace/ci-management-openstack-cron@tmp/jenkins-gitclient-ssh17961239072064067466.key
Verifying host key using manually-configured host key entries
> git fetch --tags --force --progress -- git://cloud.onap.org/mirror/ci-management +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url git://cloud.onap.org/mirror/ci-management # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision 81dc5b5d6a0624121d75e449b723b3434ba16b04 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 81dc5b5d6a0624121d75e449b723b3434ba16b04 # timeout=10
Commit message: "[POLICY] add montreal jjb jobs"
> git rev-list --no-walk 81dc5b5d6a0624121d75e449b723b3434ba16b04 # timeout=10
> git remote # timeout=10
> git submodule init # timeout=10
> git submodule sync # timeout=10
> git config --get remote.origin.url # timeout=10
> git submodule init # timeout=10
> git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10
> git config --get submodule.global-jjb.url # timeout=10
> git config -f .gitmodules --get submodule.global-jjb.path # timeout=10
> git config --get submodule.packer/common-packer.url # timeout=10
> git config -f .gitmodules --get submodule.packer/common-packer.path # timeout=10
> git submodule update --init --recursive global-jjb # timeout=10
> git submodule update --init --recursive packer/common-packer # timeout=10
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [BuilderChain]
No emails were triggered.
provisioning config files...
copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
copy managed file [npmrc] to file:/home/jenkins/.npmrc
copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins4707818051513935980.sh
---> python-tools-install.sh
Setup pyenv:
system
* 3.8.13 (set by /opt/pyenv/version)
* 3.9.13 (set by /opt/pyenv/version)
* 3.10.6 (set by /opt/pyenv/version)
lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-r6Vy
lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools
lf-activate-venv(): INFO: Adding /tmp/venv-r6Vy/bin to PATH
Generating Requirements File
Python 3.10.6
pip 23.3.1 from /tmp/venv-r6Vy/lib/python3.10/site-packages/pip (python 3.10)
appdirs==1.4.4
argcomplete==3.1.6
aspy.yaml==1.3.0
attrs==23.1.0
autopage==0.5.2
Babel==2.13.1
beautifulsoup4==4.12.2
boto3==1.29.1
botocore==1.32.1
bs4==0.0.1
cachetools==5.3.2
certifi==2023.7.22
cffi==1.16.0
cfgv==3.4.0
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
cliff==4.4.0
cmd2==2.4.3
cryptography==3.3.2
debtcollector==2.5.0
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.14
distlib==0.3.7
dnspython==2.4.2
docker==4.2.2
dogpile.cache==1.2.2
email-validator==2.1.0.post1
filelock==3.13.1
future==0.18.3
gitdb==4.0.11
GitPython==3.1.40
google-auth==2.23.4
httplib2==0.22.0
identify==2.5.31
idna==3.4
importlib-metadata==6.8.0
importlib-resources==1.5.0
iso8601==2.1.0
Jinja2==3.1.2
jmespath==1.0.1
jsonpatch==1.33
jsonpointer==2.4
jsonschema==4.19.2
jsonschema-specifications==2023.11.1
keystoneauth1==5.3.0
kubernetes==28.1.0
lftools==0.37.8
lxml==4.9.3
MarkupSafe==2.1.3
msgpack==1.0.7
multi-key-dict==2.0.3
munch==4.0.0
netaddr==0.9.0
netifaces==0.11.0
niet==1.4.2
nodeenv==1.8.0
oauth2client==4.1.3
oauthlib==3.2.2
openstacksdk==0.62.0
os-client-config==2.1.0
os-service-types==1.7.0
osc-lib==2.9.0
oslo.config==9.2.0
oslo.context==5.3.0
oslo.i18n==6.2.0
oslo.log==5.4.0
oslo.serialization==5.2.0
oslo.utils==6.3.0
packaging==23.2
pbr==6.0.0
platformdirs==3.11.0
prettytable==3.9.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pygerrit2==2.0.15
PyGithub==2.1.1
pyinotify==0.9.6
PyJWT==2.8.0
PyNaCl==1.5.0
pyparsing==2.4.7
pyperclip==1.8.2
pyrsistent==0.20.0
python-cinderclient==9.4.0
python-dateutil==2.8.2
python-heatclient==3.3.0
python-jenkins==1.8.2
python-keystoneclient==5.2.0
python-magnumclient==4.2.0
python-novaclient==18.4.0
python-openstackclient==6.0.0
python-swiftclient==4.4.0
pytz==2023.3.post1
PyYAML==6.0.1
referencing==0.31.0
requests==2.31.0
requests-oauthlib==1.3.1
requestsexceptions==1.4.0
rfc3986==2.0.0
rpds-py==0.12.0
rsa==4.9
ruamel.yaml==0.18.5
ruamel.yaml.clib==0.2.8
s3transfer==0.7.0
simplejson==3.19.2
six==1.16.0
smmap==5.0.1
soupsieve==2.5
stevedore==5.1.0
tabulate==0.9.0
toml==0.10.2
tomlkit==0.12.3
tqdm==4.66.1
typing_extensions==4.8.0
tzdata==2023.3
urllib3==1.26.18
virtualenv==20.24.6
wcwidth==0.2.10
websocket-client==1.6.4
wrapt==1.16.0
xdg==6.0.0
xmltodict==0.13.0
yq==3.2.3
zipp==3.17.0
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
OS_CLOUD=vex
[EnvInject] - Variables injected successfully.
[Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
Run condition [Boolean condition] enabling perform for step [Execute shell]
[ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins3848493921171895297.sh
---> Orphaned k8s clusters
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-r6Vy from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: kubernetes python-heatclient python-openstackclient python-magnumclient
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
lftools 0.37.8 requires openstacksdk<1.5.0, but you have openstacksdk 2.0.0 which is incompatible.
virtualenv 20.24.6 requires platformdirs<4,>=3.9.1, but you have platformdirs 4.0.0 which is incompatible.
lf-activate-venv(): INFO: Adding /tmp/venv-r6Vy/bin to PATH
-----> Delete orphaned cluster
ERROR: Failed to fetch data from https://jenkins.onap.org/sandbox/computer/api/json?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds with status code 502
502 Bad Gateway
502 Bad Gateway
nginx/1.20.0
502
Build step 'Conditional step (single)' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 21131 killed;
[ssh-agent] Stopped.
[PostBuildScript] - [INFO] Executing post build scripts.
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins3407591239365105842.sh
---> sysstat.sh
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins8684752833820576568.sh
---> package-listing.sh
++ facter osfamily
++ tr '[:upper:]' '[:lower:]'
+ OS_FAMILY=redhat
+ workspace=/w/workspace/ci-management-openstack-cron
+ START_PACKAGES=/tmp/packages_start.txt
+ END_PACKAGES=/tmp/packages_end.txt
+ DIFF_PACKAGES=/tmp/packages_diff.txt
+ PACKAGES=/tmp/packages_start.txt
+ '[' /w/workspace/ci-management-openstack-cron ']'
+ PACKAGES=/tmp/packages_end.txt
+ case "${OS_FAMILY}" in
+ rpm -qa
+ sort
+ '[' -f /tmp/packages_start.txt ']'
+ '[' -f /tmp/packages_end.txt ']'
+ diff /tmp/packages_start.txt /tmp/packages_end.txt
+ '[' /w/workspace/ci-management-openstack-cron ']'
+ mkdir -p /w/workspace/ci-management-openstack-cron/archives/
+ cp -f /tmp/packages_diff.txt /tmp/packages_end.txt /tmp/packages_start.txt /w/workspace/ci-management-openstack-cron/archives/
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins15735336500415494184.sh
---> capture-instance-metadata.sh
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-r6Vy from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools
lf-activate-venv(): INFO: Adding /tmp/venv-r6Vy/bin to PATH
INFO: Running in OpenStack, capturing instance metadata
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins16776320414182981805.sh
provisioning config files...
copy managed file [jenkins-log-archives-settings] to file:/w/workspace/ci-management-openstack-cron@tmp/config2516038796177067665tmp
Regular expression run condition: Expression=[^.*logs-s3.*], Label=[]
Run condition [Regular expression match] preventing perform for step [Provide Configuration files]
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
SERVER_ID=logs
[EnvInject] - Variables injected successfully.
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins9187217881106252906.sh
---> create-netrc.sh
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins15420933012513468742.sh
---> python-tools-install.sh
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-r6Vy from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools
lf-activate-venv(): INFO: Adding /tmp/venv-r6Vy/bin to PATH
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins278880211195940364.sh
---> sudo-logs.sh
Archiving 'sudo' log..
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins12876232753982955705.sh
---> job-cost.sh
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-r6Vy from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: zipp==1.1.0 python-openstackclient urllib3~=1.26.15
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
lftools 0.37.8 requires openstacksdk<1.5.0, but you have openstacksdk 2.0.0 which is incompatible.
virtualenv 20.24.6 requires platformdirs<4,>=3.9.1, but you have platformdirs 4.0.0 which is incompatible.
lf-activate-venv(): INFO: Adding /tmp/venv-r6Vy/bin to PATH
INFO: No Stack...
INFO: Retrieving Pricing Info for: v3-standard-2
INFO: Archiving Costs
[ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins17009586473932768353.sh
---> logs-deploy.sh
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-r6Vy from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools
lf-activate-venv(): INFO: Adding /tmp/venv-r6Vy/bin to PATH
INFO: Nexus URL https://nexus.onap.org path production/vex-yul-ecomp-jenkins-1/ci-management-openstack-cron/40603
INFO: archiving workspace using pattern(s):
Archives upload complete.
INFO: archiving logs to Nexus
---> uname -a:
Linux prd-centos8-builder-2c-1g-76326.novalocal 4.18.0-448.el8.x86_64 #1 SMP Wed Jan 18 15:02:46 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
---> lscpu:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 2
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC-Rome Processor
Stepping: 0
CPU MHz: 2799.998
BogoMIPS: 5599.99
Virtualization: AMD-V
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 16384K
NUMA node0 CPU(s): 0,1
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
---> nproc:
2
---> df -h:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.8G 0 3.8G 0% /dev/shm
tmpfs 3.8G 17M 3.8G 1% /run
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
/dev/vda1 40G 8.5G 32G 22% /
tmpfs 770M 0 770M 0% /run/user/1001
---> free -m:
total used free shared buff/cache available
Mem: 7699 562 4990 17 2146 6832
Swap: 1023 0 1023
---> ip addr:
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1458 qdisc fq_codel state UP group default qlen 1000
link/ether fa:16:3e:2e:78:ca brd ff:ff:ff:ff:ff:ff
altname enp0s3
altname ens3
inet 10.30.106.131/23 brd 10.30.107.255 scope global dynamic noprefixroute eth0
valid_lft 86226sec preferred_lft 86226sec
inet6 fe80::f816:3eff:fe2e:78ca/64 scope link
valid_lft forever preferred_lft forever
---> sar -b -r -n DEV:
Linux 4.18.0-448.el8.x86_64 (centos-stream-8-builder-63d9752a-2c92-2a07-2618-33e2bb46e6c7.nov) 11/15/2023 _x86_64_ (2 CPU)
21:03:25 LINUX RESTART (2 CPU)
09:04:01 PM tps rtps wtps bread/s bwrtn/s
09:05:01 PM 228.29 129.02 99.27 4672.18 6941.20
09:06:01 PM 104.52 0.90 103.62 103.85 10459.34
Average: 166.41 64.97 101.44 2388.20 8700.12
09:04:01 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
09:05:01 PM 5286736 7043296 2597708 32.95 2688 1928192 686940 7.69 218156 2040968 154688
09:06:01 PM 5175024 7060048 2709420 34.36 2688 2049812 666868 7.47 393344 1946056 32468
Average: 5230880 7051672 2653564 33.66 2688 1989002 676904 7.58 305750 1993512 93578
09:04:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
09:05:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:05:01 PM eth0 253.80 155.73 1741.38 61.35 0.00 0.00 0.00 0.00
09:06:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:06:01 PM eth0 42.59 32.16 416.07 10.77 0.00 0.00 0.00 0.00
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth0 148.20 93.95 1078.78 36.06 0.00 0.00 0.00 0.00
---> sar -P ALL:
Linux 4.18.0-448.el8.x86_64 (centos-stream-8-builder-63d9752a-2c92-2a07-2618-33e2bb46e6c7.nov) 11/15/2023 _x86_64_ (2 CPU)
21:03:25 LINUX RESTART (2 CPU)
09:04:01 PM CPU %user %nice %system %iowait %steal %idle
09:05:01 PM all 43.58 2.12 17.23 5.65 0.14 31.28
09:05:01 PM 0 52.23 2.11 15.97 5.34 0.15 24.20
09:05:01 PM 1 34.88 2.13 18.50 5.95 0.14 38.40
09:06:01 PM all 39.40 0.00 3.52 3.13 0.11 53.85
09:06:01 PM 0 58.33 0.00 4.02 3.19 0.12 34.33
09:06:01 PM 1 20.72 0.00 3.02 3.07 0.10 73.10
Average: all 41.48 1.05 10.34 4.38 0.13 42.62
Average: 0 55.28 1.06 10.00 4.27 0.14 29.26
Average: 1 27.73 1.05 10.68 4.50 0.12 55.92