15:04:50  Started by timer
15:04:50  Running as SYSTEM
15:04:50  [EnvInject] - Loading node environment variables.
15:04:50  Building remotely on prd-centos8-builder-2c-1g-6977 (centos8-builder-2c-1g) in workspace /w/workspace/ci-management-openstack-cron
15:04:51  [ssh-agent] Looking for ssh-agent implementation...
15:04:51  [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
15:04:51  $ ssh-agent
15:04:51  SSH_AUTH_SOCK=/tmp/ssh-ecK7Wp9y5Gap/agent.15130
15:04:51  SSH_AGENT_PID=15152
15:04:51  [ssh-agent] Started.
15:04:51  Running ssh-add (command line suppressed)
15:04:51  Identity added: /w/workspace/ci-management-openstack-cron@tmp/private_key_17199060627089016671.key (/w/workspace/ci-management-openstack-cron@tmp/private_key_17199060627089016671.key)
15:04:51  [ssh-agent] Using credentials onap-jobbuiler (Gerrit user)
15:04:51  The recommended git tool is: NONE
15:04:53  using credential onap-jenkins-ssh
15:04:53  Wiping out workspace first.
15:04:53  Cloning the remote Git repository
15:04:53  Cloning repository git://cloud.onap.org/mirror/ci-management
15:04:53   > git init /w/workspace/ci-management-openstack-cron # timeout=10
15:04:53  Fetching upstream changes from git://cloud.onap.org/mirror/ci-management
15:04:53   > git --version # timeout=10
15:04:53   > git --version # 'git version 2.39.0'
15:04:53  using GIT_SSH to set credentials Gerrit user
15:04:53  [INFO] Currently running in a labeled security context
15:04:53  [INFO] Currently SELinux is 'enforcing' on the host
15:04:53   > /usr/bin/chcon --type=ssh_home_t /w/workspace/ci-management-openstack-cron@tmp/jenkins-gitclient-ssh5165083813867464997.key
15:04:53  Verifying host key using manually-configured host key entries
15:04:53   > git fetch --tags --force --progress -- git://cloud.onap.org/mirror/ci-management +refs/heads/*:refs/remotes/origin/* # timeout=10
15:04:54   > git config remote.origin.url git://cloud.onap.org/mirror/ci-management # timeout=10
15:04:54   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
15:04:55  Avoid second fetch
15:04:55   > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
15:04:55  Checking out Revision 5202ccaf89a0896a7f3f43a2e2705eccb960d7f4 (refs/remotes/origin/master)
15:04:55   > git config core.sparsecheckout # timeout=10
15:04:55   > git checkout -f 5202ccaf89a0896a7f3f43a2e2705eccb960d7f4 # timeout=10
15:04:56  Commit message: "Use larger build node for multicloud-k8s-verify job"
15:04:56   > git rev-list --no-walk 5202ccaf89a0896a7f3f43a2e2705eccb960d7f4 # timeout=10
15:04:56   > git remote # timeout=10
15:04:56   > git submodule init # timeout=10
15:04:56   > git submodule sync # timeout=10
15:04:56   > git config --get remote.origin.url # timeout=10
15:04:56   > git submodule init # timeout=10
15:04:56   > git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10
15:04:56   > git config --get submodule.global-jjb.url # timeout=10
15:04:56   > git config -f .gitmodules --get submodule.global-jjb.path # timeout=10
15:04:56   > git config --get submodule.packer/common-packer.url # timeout=10
15:04:56   > git config -f .gitmodules --get submodule.packer/common-packer.path # timeout=10
15:04:56   > git submodule update --init --recursive global-jjb # timeout=10
15:04:58   > git submodule update --init --recursive packer/common-packer # timeout=10
15:05:01  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
15:05:01  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
15:05:01  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
15:05:01  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
15:05:01  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
15:05:01  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
15:05:01  Run condition [Boolean condition] enabling prebuild for step [BuilderChain]
15:05:01  No emails were triggered.
15:05:01  provisioning config files...
15:05:02  copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
15:05:02  copy managed file [npmrc] to file:/home/jenkins/.npmrc
15:05:02  copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf
15:05:02  [ci-management-openstack-cron] $ /bin/bash /tmp/jenkins768463645900248403.sh
15:05:02  ---> python-tools-install.sh
15:05:02  Setup pyenv:
15:05:02    system
15:05:02  * 3.8.13 (set by /opt/pyenv/version)
15:05:02  * 3.9.13 (set by /opt/pyenv/version)
15:05:02  * 3.10.6 (set by /opt/pyenv/version)
15:05:06  lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-VMOO
15:05:06  lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv
15:05:06  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:05:06  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:05:11  lf-activate-venv(): INFO: Base packages installed successfully
15:05:11  lf-activate-venv(): INFO: Installing additional packages: lftools 
15:05:38  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:05:38  Generating Requirements File
15:05:57  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
15:05:57  deprecated 1.2.18 requires wrapt<2,>=1.10, but you have wrapt 2.0.0 which is incompatible.
15:05:57  httplib2 0.31.0 requires pyparsing<4,>=3.0.4, but you have pyparsing 2.4.7 which is incompatible.
15:05:58  Python 3.10.6
15:05:58  pip 25.3 from /tmp/venv-VMOO/lib/python3.10/site-packages/pip (python 3.10)
15:05:58  appdirs==1.4.4
15:05:58  argcomplete==3.6.3
15:05:58  aspy.yaml==1.3.0
15:05:58  attrs==25.4.0
15:05:58  autopage==0.5.2
15:05:58  beautifulsoup4==4.14.2
15:05:58  boto3==1.40.60
15:05:58  botocore==1.40.60
15:05:58  bs4==0.0.2
15:05:58  cachetools==6.2.1
15:05:58  certifi==2025.10.5
15:05:58  cffi==2.0.0
15:05:58  cfgv==3.4.0
15:05:58  chardet==5.2.0
15:05:58  charset-normalizer==3.4.4
15:05:58  click==8.3.0
15:05:58  cliff==4.11.0
15:05:58  cmd2==2.7.0
15:05:58  cryptography==3.3.2
15:05:58  debtcollector==3.0.0
15:05:58  decorator==5.2.1
15:05:58  defusedxml==0.7.1
15:05:58  Deprecated==1.2.18
15:05:58  distlib==0.4.0
15:05:58  dnspython==2.8.0
15:05:58  docker==7.1.0
15:05:58  dogpile.cache==1.5.0
15:05:58  durationpy==0.10
15:05:58  email-validator==2.3.0
15:05:58  filelock==3.20.0
15:05:58  future==1.0.0
15:05:58  gitdb==4.0.12
15:05:58  GitPython==3.1.45
15:05:58  google-auth==2.41.1
15:05:58  httplib2==0.31.0
15:05:58  identify==2.6.15
15:05:58  idna==3.11
15:05:58  importlib-resources==1.5.0
15:05:58  iso8601==2.1.0
15:05:58  Jinja2==3.1.6
15:05:58  jmespath==1.0.1
15:05:58  jsonpatch==1.33
15:05:58  jsonpointer==3.0.0
15:05:58  jsonschema==4.25.1
15:05:58  jsonschema-specifications==2025.9.1
15:05:58  keystoneauth1==5.12.0
15:05:58  kubernetes==34.1.0
15:05:58  lftools==0.37.15
15:05:58  lxml==6.0.2
15:05:58  markdown-it-py==4.0.0
15:05:58  MarkupSafe==3.0.3
15:05:58  mdurl==0.1.2
15:05:58  msgpack==1.1.2
15:05:58  multi_key_dict==2.0.3
15:05:58  munch==4.0.0
15:05:58  netaddr==1.3.0
15:05:58  niet==1.4.2
15:05:58  nodeenv==1.9.1
15:05:58  oauth2client==4.1.3
15:05:58  oauthlib==3.3.1
15:05:58  openstacksdk==4.7.1
15:05:58  os-service-types==1.8.0
15:05:58  osc-lib==4.2.0
15:05:58  oslo.config==10.0.0
15:05:58  oslo.context==6.1.0
15:05:58  oslo.i18n==6.6.0
15:05:58  oslo.log==7.2.1
15:05:58  oslo.serialization==5.8.0
15:05:58  oslo.utils==9.1.0
15:05:58  packaging==25.0
15:05:58  pbr==7.0.1
15:05:58  platformdirs==4.5.0
15:05:58  prettytable==3.16.0
15:05:58  psutil==7.1.2
15:05:58  pyasn1==0.6.1
15:05:58  pyasn1_modules==0.4.2
15:05:58  pycparser==2.23
15:05:58  pygerrit2==2.0.15
15:05:58  PyGithub==2.8.1
15:05:58  Pygments==2.19.2
15:05:58  PyJWT==2.10.1
15:05:58  PyNaCl==1.6.0
15:05:58  pyparsing==2.4.7
15:05:58  pyperclip==1.11.0
15:05:58  pyrsistent==0.20.0
15:05:58  python-cinderclient==9.8.0
15:05:58  python-dateutil==2.9.0.post0
15:05:58  python-heatclient==4.3.0
15:05:58  python-jenkins==1.8.3
15:05:58  python-keystoneclient==5.7.0
15:05:58  python-magnumclient==4.9.0
15:05:58  python-openstackclient==8.2.0
15:05:58  python-swiftclient==4.8.0
15:05:58  PyYAML==6.0.3
15:05:58  referencing==0.37.0
15:05:58  requests==2.32.5
15:05:58  requests-oauthlib==2.0.0
15:05:58  requestsexceptions==1.4.0
15:05:58  rfc3986==2.0.0
15:05:58  rich==14.2.0
15:05:58  rich-argparse==1.7.1
15:05:58  rpds-py==0.28.0
15:05:58  rsa==4.9.1
15:05:58  ruamel.yaml==0.18.16
15:05:58  ruamel.yaml.clib==0.2.14
15:05:58  s3transfer==0.14.0
15:05:58  simplejson==3.20.2
15:05:58  six==1.17.0
15:05:58  smmap==5.0.2
15:05:58  soupsieve==2.8
15:05:58  stevedore==5.5.0
15:05:58  tabulate==0.9.0
15:05:58  toml==0.10.2
15:05:58  tomlkit==0.13.3
15:05:58  tqdm==4.67.1
15:05:58  typing_extensions==4.15.0
15:05:58  tzdata==2025.2
15:05:58  urllib3==1.26.20
15:05:58  virtualenv==20.35.3
15:05:58  wcwidth==0.2.14
15:05:58  websocket-client==1.9.0
15:05:58  wrapt==2.0.0
15:05:58  xdg==6.0.0
15:05:58  xmltodict==1.0.2
15:05:58  yq==3.4.3
15:05:58  [EnvInject] - Injecting environment variables from a build step.
15:05:58  [EnvInject] - Injecting as environment variables the properties content 
15:05:58  OS_CLOUD=vex
15:05:58  
15:05:58  [EnvInject] - Variables injected successfully.
15:05:58  [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
15:05:58  Run condition [Boolean condition] enabling perform for step [Execute shell]
15:05:58  [ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins16316169458018118317.sh
15:05:58  ---> Orphaned k8s clusters
15:05:58  Setup pyenv:
15:05:59    system
15:05:59    3.8.13
15:05:59    3.9.13
15:05:59  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:05:59  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:05:59  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:05:59  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:06:01  lf-activate-venv(): INFO: Base packages installed successfully
15:06:01  lf-activate-venv(): INFO: Installing additional packages: kubernetes python-heatclient python-openstackclient python-magnumclient 
15:06:07  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
15:06:07  lftools 0.37.15 requires urllib3<2.1.0, but you have urllib3 2.3.0 which is incompatible.
15:06:07  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:06:10  -----> Delete orphaned cluster
15:06:10  [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
15:06:10  Run condition [Boolean condition] enabling perform for step [Execute shell]
15:06:10  [ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins2896699805435005553.sh
15:06:10  ---> Orphaned stacks
15:06:10  Setup pyenv:
15:06:11    system
15:06:11    3.8.13
15:06:11    3.9.13
15:06:11  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:06:11  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:06:11  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:06:11  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:06:13  lf-activate-venv(): INFO: Base packages installed successfully
15:06:13  lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq 
15:06:26  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:06:26  + mapfile -t OS_COE_CLUSTERS_ID
15:06:26  ++ openstack --os-cloud vex coe cluster list -f value -c uuid -c name
15:06:26  ++ grep -E '(DELETE_FAILED|UNKNOWN|UNHEALTHY)'
15:06:26  ++ awk '{print $1}'
15:06:28  + echo '-----> Active clusters -> stacks'
15:06:28  -----> Active clusters -> stacks
15:06:28  + OS_COE_STACKS=()
15:06:28  + [[ 0 -gt 0 ]]
15:06:28  + mapfile -t OS_STACKS
15:06:28  ++ openstack --os-cloud vex stack list -f value -c 'Stack Name' -c 'Stack Status' --property stack_status=CREATE_COMPLETE --property stack_status=DELETE_FAILED --property stack_status=CREATE_FAILED
15:06:28  ++ awk '{print $1}'
15:06:29  + echo '-----> Active stacks'
15:06:29  -----> Active stacks
15:06:29  + echo '-----> Delete orphaned stacks'
15:06:29  -----> Delete orphaned stacks
15:06:29  [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
15:06:29  Run condition [Boolean condition] enabling perform for step [Execute shell]
15:06:29  [ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins18085114061619399571.sh
15:06:29  ---> Orphaned servers
15:06:29  Setup pyenv:
15:06:29    system
15:06:29    3.8.13
15:06:29    3.9.13
15:06:29  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:06:30  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:06:30  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:06:30  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:06:32  lf-activate-venv(): INFO: Base packages installed successfully
15:06:32  lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq 
15:06:43  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:06:46  -----> Active servers
15:06:46  prd-centos8-builder-2c-1g-6977
15:06:46  prd-ubuntu1804-docker-8c-8g-6976
15:06:46  prd-ubuntu2204-docker-8c-8g-6975
15:06:46  prd-queue-docker-4c-4g-6180
15:06:46  prd-ubuntu2004-helm-2c-1g-6006
15:06:46  -----> Delete orphaned servers
15:06:47  [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
15:06:47  Run condition [Boolean condition] enabling perform for step [Execute shell]
15:06:47  [ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins13824583198161983455.sh
15:06:47  ---> Orphaned ports
15:06:47  Setup pyenv:
15:06:47    system
15:06:47    3.8.13
15:06:47    3.9.13
15:06:47  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:06:47  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:06:47  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:06:47  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:06:49  lf-activate-venv(): INFO: Base packages installed successfully
15:06:49  lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] python-openstackclient 
15:07:00  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:07:02  No orphaned ports to process.
15:07:02  [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
15:07:02  Run condition [Boolean condition] enabling perform for step [Execute shell]
15:07:02  [ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins18372937066877425947.sh
15:07:02  ---> Orphaned volumes
15:07:02  + source /home/jenkins/lf-env.sh
15:07:02  + lf-activate-venv --python python3 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
15:07:02  ++ mktemp -d /tmp/venv-XXXX
15:07:02  + lf_venv=/tmp/venv-QBrp
15:07:02  + local venv_file=/tmp/.os_lf_venv
15:07:02  + local python=python3
15:07:02  + local options
15:07:02  + local set_path=true
15:07:02  + local install_args=
15:07:02  ++ getopt -o np:v: -l no-path,system-site-packages,python:,venv-file: -n lf-activate-venv -- --python python3 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
15:07:02  + options=' --python '\''python3'\'' -- '\''lftools[openstack]'\'' '\''kubernetes'\'' '\''niet'\'' '\''python-heatclient'\'' '\''python-openstackclient'\'' '\''python-magnumclient'\'' '\''yq'\'''
15:07:02  + eval set -- ' --python '\''python3'\'' -- '\''lftools[openstack]'\'' '\''kubernetes'\'' '\''niet'\'' '\''python-heatclient'\'' '\''python-openstackclient'\'' '\''python-magnumclient'\'' '\''yq'\'''
15:07:02  ++ set -- --python python3 -- 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
15:07:02  + true
15:07:02  + case $1 in
15:07:02  + python=python3
15:07:02  + shift 2
15:07:02  + true
15:07:02  + case $1 in
15:07:02  + shift
15:07:02  + break
15:07:02  + case $python in
15:07:02  + local pkg_list=
15:07:02  + [[ -d /opt/pyenv ]]
15:07:02  + echo 'Setup pyenv:'
15:07:02  Setup pyenv:
15:07:02  + export PYENV_ROOT=/opt/pyenv
15:07:02  + PYENV_ROOT=/opt/pyenv
15:07:02  + export PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
15:07:02  + PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
15:07:02  + pyenv versions
15:07:02    system
15:07:02    3.8.13
15:07:02    3.9.13
15:07:02  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:07:02  + command -v pyenv
15:07:02  ++ pyenv init - --no-rehash
15:07:02  + eval 'PATH="$(bash --norc -ec '\''IFS=:; paths=($PATH); for i in ${!paths[@]}; do if [[ ${paths[i]} == "'\''/opt/pyenv/shims'\''" ]]; then unset '\''\'\'''\''paths[i]'\''\'\'''\''; fi; done; echo "${paths[*]}"'\'')"
15:07:02  export PATH="/opt/pyenv/shims:${PATH}"
15:07:02  export PYENV_SHELL=bash
15:07:02  source '\''/opt/pyenv/libexec/../completions/pyenv.bash'\''
15:07:02  pyenv() {
15:07:02    local command
15:07:02    command="${1:-}"
15:07:02    if [ "$#" -gt 0 ]; then
15:07:02      shift
15:07:02    fi
15:07:02  
15:07:02    case "$command" in
15:07:02    rehash|shell)
15:07:02      eval "$(pyenv "sh-$command" "$@")"
15:07:02      ;;
15:07:02    *)
15:07:02      command pyenv "$command" "$@"
15:07:02      ;;
15:07:02    esac
15:07:02  }'
15:07:02  +++ bash --norc -ec 'IFS=:; paths=($PATH); for i in ${!paths[@]}; do if [[ ${paths[i]} == "/opt/pyenv/shims" ]]; then unset '\''paths[i]'\''; fi; done; echo "${paths[*]}"'
15:07:02  ++ PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
15:07:02  ++ export PATH=/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
15:07:02  ++ PATH=/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
15:07:02  ++ export PYENV_SHELL=bash
15:07:02  ++ PYENV_SHELL=bash
15:07:02  ++ source /opt/pyenv/libexec/../completions/pyenv.bash
15:07:02  +++ complete -F _pyenv pyenv
15:07:02  ++ lf-pyver python3
15:07:02  ++ local py_version_xy=python3
15:07:02  ++ local py_version_xyz=
15:07:02  ++ awk '{ print $1 }'
15:07:02  ++ pyenv versions
15:07:02  ++ local command
15:07:02  ++ command=versions
15:07:02  ++ '[' 1 -gt 0 ']'
15:07:02  ++ shift
15:07:02  ++ case "$command" in
15:07:02  ++ command pyenv versions
15:07:02  ++ pyenv versions
15:07:02  ++ sed 's/^[ *]* //'
15:07:02  ++ grep -E '^[0-9.]*[0-9]$'
15:07:03  ++ [[ ! -s /tmp/.pyenv_versions ]]
15:07:03  +++ tail -n 1
15:07:03  +++ grep '^3' /tmp/.pyenv_versions
15:07:03  +++ sort -V
15:07:03  ++ py_version_xyz=3.10.6
15:07:03  ++ [[ -z 3.10.6 ]]
15:07:03  ++ echo 3.10.6
15:07:03  ++ return 0
15:07:03  + pyenv local 3.10.6
15:07:03  + local command
15:07:03  + command=local
15:07:03  + '[' 2 -gt 0 ']'
15:07:03  + shift
15:07:03  + case "$command" in
15:07:03  + command pyenv local 3.10.6
15:07:03  + pyenv local 3.10.6
15:07:03  + for arg in "$@"
15:07:03  + case $arg in
15:07:03  + pkg_list+='lftools[openstack] '
15:07:03  + for arg in "$@"
15:07:03  + case $arg in
15:07:03  + pkg_list+='kubernetes '
15:07:03  + for arg in "$@"
15:07:03  + case $arg in
15:07:03  + pkg_list+='niet '
15:07:03  + for arg in "$@"
15:07:03  + case $arg in
15:07:03  + pkg_list+='python-heatclient '
15:07:03  + for arg in "$@"
15:07:03  + case $arg in
15:07:03  + pkg_list+='python-openstackclient '
15:07:03  + for arg in "$@"
15:07:03  + case $arg in
15:07:03  + pkg_list+='python-magnumclient '
15:07:03  + for arg in "$@"
15:07:03  + case $arg in
15:07:03  + pkg_list+='yq '
15:07:03  + [[ -f /tmp/.os_lf_venv ]]
15:07:03  ++ cat /tmp/.os_lf_venv
15:07:03  + lf_venv=/tmp/venv-VMOO
15:07:03  + echo 'lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from' file:/tmp/.os_lf_venv
15:07:03  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:07:03  + echo 'lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)'
15:07:03  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:07:03  + local 'pip_opts=--upgrade --quiet'
15:07:03  + pip_opts='--upgrade --quiet --trusted-host pypi.org'
15:07:03  + pip_opts='--upgrade --quiet --trusted-host pypi.org --trusted-host files.pythonhosted.org'
15:07:03  + pip_opts='--upgrade --quiet --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org'
15:07:03  + [[ -n '' ]]
15:07:03  + [[ -n '' ]]
15:07:03  + echo 'lf-activate-venv(): INFO: Attempting to install with network-safe options...'
15:07:03  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:07:03  + /tmp/venv-VMOO/bin/python3 -m pip install --upgrade --quiet --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org pip 'setuptools<66' virtualenv
15:07:04  + echo 'lf-activate-venv(): INFO: Base packages installed successfully'
15:07:04  lf-activate-venv(): INFO: Base packages installed successfully
15:07:04  + [[ -z lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq  ]]
15:07:04  + echo 'lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq '
15:07:04  lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq 
15:07:04  + /tmp/venv-VMOO/bin/python3 -m pip install --upgrade --quiet --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org --upgrade-strategy eager 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
15:07:22  + type python3
15:07:22  + true
15:07:22  + echo 'lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH'
15:07:22  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:07:22  + PATH=/tmp/venv-VMOO/bin:/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
15:07:22  + return 0
15:07:22  + mapfile -t os_volumes
15:07:22  ++ openstack --os-cloud vex volume list -f value -c ID --status Available
15:07:25  + '[' 0 -eq 0 ']'
15:07:25  + echo 'No orphaned volumes found.'
15:07:25  No orphaned volumes found.
15:07:25  [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
15:07:25  Run condition [Boolean condition] enabling perform for step [Execute shell]
15:07:25  [ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins3522028887772266335.sh
15:07:25  ---> Protect in-use images
15:07:25  Setup pyenv:
15:07:25    system
15:07:25    3.8.13
15:07:25    3.9.13
15:07:25  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:07:25  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:07:25  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:07:25  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:07:27  lf-activate-venv(): INFO: Base packages installed successfully
15:07:27  lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] kubernetes python-heatclient python-openstackclient 
15:07:39  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:07:39  INFO: There are images to protect defined in jenkins-config.
15:07:39  INFO: Protecting the following images:
15:07:39  CentOS Stream 8 - redis -  - 310180101-1010
15:07:39  Ubuntu 20.04 - memcached - x86_64 - 160580505-5011
15:07:39  ZZCI - CentOS 7 - builder - x86_64 - 20211001-132245.747
15:07:39  ZZCI - CentOS 7 - docker - x86_64 - 20220919-201912.261
15:07:39  ZZCI - CentOS Stream 8 - builder - x86_64 - 20230131-200810.126
15:07:39  ZZCI - CentOS Stream 8 - docker - x86_64 - 20230215-011800.297
15:07:39  ZZCI - Ubuntu 18.04 - builder - x86_64 - 20221013-041722.509
15:07:39  ZZCI - Ubuntu 18.04 - docker - x86_64 - 20220919-185927.020
15:07:39  ZZCI - Ubuntu 20.04 - builder - x86_64 - 20230412-164405.694
15:07:39  ZZCI - Ubuntu 20.04 - devstack - x86_64 - 20230913-103335.070
15:07:39  ZZCI - Ubuntu 20.04 - docker - x86_64 - 20220919-201946.022
15:07:39  ZZCI - Ubuntu 20.04 - docker - x86_64 - 20230414-200946.220
15:07:39  ZZCI - Ubuntu 20.04 - helm - x86_64 - 20230405-210810.069
15:07:39  ZZCI - Ubuntu 22.04 - builder - x86_64 - 20250424-093435.248
15:07:39  ZZCI - Ubuntu 22.04 - docker - x86_64 - 20250623-071957.883
15:07:41  Protected setting for CentOS Stream 8 - redis -  - 310180101-1010: True
15:07:42  Protected setting for Ubuntu 20.04 - memcached - x86_64 - 160580505-5011: True
15:07:44  No Image found for ZZCI - CentOS 7 - builder - x86_64 - 20211001-132245.747
15:07:44  Build step 'Conditional step (single)' marked build as failure
15:07:44  $ ssh-agent -k
15:07:44  unset SSH_AUTH_SOCK;
15:07:44  unset SSH_AGENT_PID;
15:07:44  echo Agent pid 15152 killed;
15:07:45  [ssh-agent] Stopped.
15:07:45  [PostBuildScript] - [INFO] Executing post build scripts.
15:07:45  [ci-management-openstack-cron] $ /bin/bash /tmp/jenkins2479822714071872786.sh
15:07:45  ---> sysstat.sh
15:07:45  [ci-management-openstack-cron] $ /bin/bash /tmp/jenkins1837620279646190511.sh
15:07:45  ---> package-listing.sh
15:07:45  ++ facter osfamily
15:07:45  ++ tr '[:upper:]' '[:lower:]'
15:07:45  + OS_FAMILY=redhat
15:07:45  + workspace=/w/workspace/ci-management-openstack-cron
15:07:45  + START_PACKAGES=/tmp/packages_start.txt
15:07:45  + END_PACKAGES=/tmp/packages_end.txt
15:07:45  + DIFF_PACKAGES=/tmp/packages_diff.txt
15:07:45  + PACKAGES=/tmp/packages_start.txt
15:07:45  + '[' /w/workspace/ci-management-openstack-cron ']'
15:07:45  + PACKAGES=/tmp/packages_end.txt
15:07:45  + case "${OS_FAMILY}" in
15:07:45  + rpm -qa
15:07:45  + sort
15:07:50  + '[' -f /tmp/packages_start.txt ']'
15:07:50  + '[' -f /tmp/packages_end.txt ']'
15:07:50  + diff /tmp/packages_start.txt /tmp/packages_end.txt
15:07:50  + '[' /w/workspace/ci-management-openstack-cron ']'
15:07:50  + mkdir -p /w/workspace/ci-management-openstack-cron/archives/
15:07:50  + cp -f /tmp/packages_diff.txt /tmp/packages_end.txt /tmp/packages_start.txt /w/workspace/ci-management-openstack-cron/archives/
15:07:50  [ci-management-openstack-cron] $ /bin/bash /tmp/jenkins16696773079958226730.sh
15:07:50  ---> capture-instance-metadata.sh
15:07:50  Setup pyenv:
15:07:50    system
15:07:50    3.8.13
15:07:50    3.9.13
15:07:50  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:07:50  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:07:50  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:07:50  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:07:52  lf-activate-venv(): INFO: Base packages installed successfully
15:07:52  lf-activate-venv(): INFO: Installing additional packages: lftools 
15:08:00  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:08:00  INFO: Running in OpenStack, capturing instance metadata
15:08:01  [ci-management-openstack-cron] $ /bin/bash /tmp/jenkins15121641621029785556.sh
15:08:01  provisioning config files...
15:08:01  copy managed file [jenkins-log-archives-settings] to file:/w/workspace/ci-management-openstack-cron@tmp/config13708471683469717931tmp
15:08:01  Regular expression run condition: Expression=[^.*logs-s3.*], Label=[]
15:08:01  Run condition [Regular expression match] preventing perform for step [Provide Configuration files]
15:08:01  [EnvInject] - Injecting environment variables from a build step.
15:08:01  [EnvInject] - Injecting as environment variables the properties content 
15:08:01  SERVER_ID=logs
15:08:01  
15:08:01  [EnvInject] - Variables injected successfully.
15:08:01  [ci-management-openstack-cron] $ /bin/bash /tmp/jenkins3399136780913709158.sh
15:08:01  ---> create-netrc.sh
15:08:01  [ci-management-openstack-cron] $ /bin/bash /tmp/jenkins8817776350592524222.sh
15:08:01  ---> python-tools-install.sh
15:08:01  Setup pyenv:
15:08:01    system
15:08:01    3.8.13
15:08:01    3.9.13
15:08:01  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:08:02  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:08:02  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:08:02  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:08:03  lf-activate-venv(): INFO: Base packages installed successfully
15:08:03  lf-activate-venv(): INFO: Installing additional packages: lftools 
15:08:11  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:08:11  [ci-management-openstack-cron] $ /bin/bash /tmp/jenkins10368183172501515890.sh
15:08:11  ---> sudo-logs.sh
15:08:11  Archiving 'sudo' log..
15:08:12  [ci-management-openstack-cron] $ /bin/bash /tmp/jenkins6038309707776734344.sh
15:08:12  ---> job-cost.sh
15:08:12  Setup pyenv:
15:08:12    system
15:08:12    3.8.13
15:08:12    3.9.13
15:08:12  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:08:12  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:08:12  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:08:12  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:08:14  lf-activate-venv(): INFO: Base packages installed successfully
15:08:14  lf-activate-venv(): INFO: Installing additional packages: zipp==1.1.0 python-openstackclient urllib3~=1.26.15 
15:08:20  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
15:08:20  deprecated 1.2.18 requires wrapt<2,>=1.10, but you have wrapt 2.0.0 which is incompatible.
15:08:20  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:08:20  INFO: No Stack...
15:08:20  INFO: Retrieving Pricing Info for: v3-standard-2
15:08:21  INFO: Archiving Costs
15:08:21  [ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins17159738343492908241.sh
15:08:21  ---> logs-deploy.sh
15:08:21  Setup pyenv:
15:08:21    system
15:08:21    3.8.13
15:08:21    3.9.13
15:08:21  * 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
15:08:21  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-VMOO from file:/tmp/.os_lf_venv
15:08:21  lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)
15:08:21  lf-activate-venv(): INFO: Attempting to install with network-safe options...
15:08:23  lf-activate-venv(): INFO: Base packages installed successfully
15:08:23  lf-activate-venv(): INFO: Installing additional packages: lftools 
15:08:31  lf-activate-venv(): INFO: Adding /tmp/venv-VMOO/bin to PATH
15:08:31  INFO: Nexus URL https://nexus.onap.org path production/vex-yul-ecomp-jenkins-1/ci-management-openstack-cron/57672
15:08:31  INFO: archiving workspace using pattern(s): 
15:08:32  Archives upload complete.
15:08:32  INFO: archiving logs to Nexus
15:08:33  ---> uname -a:
15:08:33  Linux prd-centos8-builder-2c-1g-6977.novalocal 4.18.0-448.el8.x86_64 #1 SMP Wed Jan 18 15:02:46 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
15:08:33  
15:08:33  
15:08:33  ---> lscpu:
15:08:33  Architecture:        x86_64
15:08:33  CPU op-mode(s):      32-bit, 64-bit
15:08:33  Byte Order:          Little Endian
15:08:33  CPU(s):              2
15:08:33  On-line CPU(s) list: 0,1
15:08:33  Thread(s) per core:  1
15:08:33  Core(s) per socket:  1
15:08:33  Socket(s):           2
15:08:33  NUMA node(s):        1
15:08:33  Vendor ID:           AuthenticAMD
15:08:33  CPU family:          23
15:08:33  Model:               49
15:08:33  Model name:          AMD EPYC-Rome Processor
15:08:33  Stepping:            0
15:08:33  CPU MHz:             2800.000
15:08:33  BogoMIPS:            5600.00
15:08:33  Virtualization:      AMD-V
15:08:33  Hypervisor vendor:   KVM
15:08:33  Virtualization type: full
15:08:33  L1d cache:           32K
15:08:33  L1i cache:           32K
15:08:33  L2 cache:            512K
15:08:33  L3 cache:            16384K
15:08:33  NUMA node0 CPU(s):   0,1
15:08:33  Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
15:08:33  
15:08:33  
15:08:33  ---> nproc:
15:08:33  2
15:08:33  
15:08:33  
15:08:33  ---> df -h:
15:08:33  Filesystem      Size  Used Avail Use% Mounted on
15:08:33  devtmpfs        3.8G     0  3.8G   0% /dev
15:08:33  tmpfs           3.8G     0  3.8G   0% /dev/shm
15:08:33  tmpfs           3.8G   17M  3.8G   1% /run
15:08:33  tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
15:08:33  /dev/vda1        40G  8.4G   32G  21% /
15:08:33  tmpfs           770M     0  770M   0% /run/user/1001
15:08:33  
15:08:33  
15:08:33  ---> free -m:
15:08:33                total        used        free      shared  buff/cache   available
15:08:33  Mem:           7699         530        5074          19        2095        6862
15:08:33  Swap:          1023           0        1023
15:08:33  
15:08:33  
15:08:33  ---> ip addr:
15:08:33  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
15:08:33      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
15:08:33      inet 127.0.0.1/8 scope host lo
15:08:33         valid_lft forever preferred_lft forever
15:08:33      inet6 ::1/128 scope host 
15:08:33         valid_lft forever preferred_lft forever
15:08:33  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc mq state UP group default qlen 1000
15:08:33      link/ether fa:16:3e:ad:0a:94 brd ff:ff:ff:ff:ff:ff
15:08:33      altname enp0s3
15:08:33      altname ens3
15:08:33      inet 10.30.106.95/23 brd 10.30.107.255 scope global dynamic noprefixroute eth0
15:08:33         valid_lft 86146sec preferred_lft 86146sec
15:08:33      inet6 fe80::f816:3eff:fead:a94/64 scope link 
15:08:33         valid_lft forever preferred_lft forever
15:08:33  
15:08:33  
15:08:33  ---> sar -b -r -n DEV:
15:08:33  Linux 4.18.0-448.el8.x86_64 (centos-stream-8-builder-63d9752a-2c92-2a07-2618-33e2bb46e6c7.nov) 	10/28/2025 	_x86_64_	(2 CPU)
15:08:33  
15:08:33  15:04:15     LINUX RESTART	(2 CPU)
15:08:33  
15:08:33  03:05:03 PM       tps      rtps      wtps   bread/s   bwrtn/s
15:08:33  03:06:01 PM    214.26     95.47    118.79   1147.94   7883.35
15:08:33  03:07:01 PM     94.28      0.72     93.57     94.52   9038.28
15:08:33  03:08:01 PM     25.77      0.17     25.61     12.26    920.46
15:08:33  Average:       110.28     31.40     78.88    410.02   5925.33
15:08:33  
15:08:33  03:05:03 PM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
15:08:33  03:06:01 PM   5282256   7095708   2602188     33.00      2688   1980132    607496      6.80    196188   2042620    220504
15:08:33  03:07:01 PM   5248984   7076188   2635460     33.43      2688   1992720    617392      6.91    225852   2034176      8844
15:08:33  03:08:01 PM   5267644   7095144   2616800     33.19      2688   1993236    601796      6.74    396044   1846772      8856
15:08:33  Average:      5266295   7089013   2618149     33.21      2688   1988696    608895      6.82    272695   1974523     79401
15:08:33  
15:08:33  03:05:03 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
15:08:33  03:06:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
15:08:33  03:06:01 PM      eth0     74.59     53.06   1172.06      9.60      0.00      0.00      0.00      0.00
15:08:33  03:07:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
15:08:33  03:07:01 PM      eth0     25.93     23.26     48.51      8.65      0.00      0.00      0.00      0.00
15:08:33  03:08:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
15:08:33  03:08:01 PM      eth0     26.41     21.43     20.37     12.33      0.00      0.00      0.00      0.00
15:08:33  Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
15:08:33  Average:         eth0     41.95     32.35    405.11     10.20      0.00      0.00      0.00      0.00
15:08:33  
15:08:33  
15:08:33  ---> sar -P ALL:
15:08:33  Linux 4.18.0-448.el8.x86_64 (centos-stream-8-builder-63d9752a-2c92-2a07-2618-33e2bb46e6c7.nov) 	10/28/2025 	_x86_64_	(2 CPU)
15:08:33  
15:08:33  15:04:15     LINUX RESTART	(2 CPU)
15:08:33  
15:08:33  03:05:03 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
15:08:33  03:06:01 PM     all     39.21      1.58     15.96      5.48      0.13     37.63
15:08:33  03:06:01 PM       0     35.61      1.23     13.12      4.68      0.12     45.24
15:08:33  03:06:01 PM       1     42.87      1.93     18.84      6.30      0.14     29.91
15:08:33  03:07:01 PM     all     39.98      0.00      3.10      2.01      0.11     54.80
15:08:33  03:07:01 PM       0     40.48      0.00      2.88      1.16      0.12     55.36
15:08:33  03:07:01 PM       1     39.47      0.00      3.32      2.86      0.10     54.25
15:08:33  03:08:01 PM     all     32.77      0.00      2.27      0.90      0.12     63.94
15:08:33  03:08:01 PM       0     20.47      0.00      1.95      0.18      0.13     77.27
15:08:33  03:08:01 PM       1     45.08      0.00      2.58      1.62      0.12     50.60
15:08:33  Average:        all     37.28      0.50      6.89      2.73      0.12     52.47
15:08:33  Average:          0     32.12      0.39      5.82      1.95      0.12     59.59
15:08:33  Average:          1     42.46      0.61      7.97      3.52      0.12     45.32
15:08:33  
15:08:33  
15:08:33