By type
Configuring logger redirection
2021-05-03 23:33:03.189 [debug] <0.288.0> Lager installed handler error_logger_lager_h into error_logger
2021-05-03 23:33:03.220 [debug] <0.291.0> Lager installed handler lager_forwarder_backend into error_logger_lager_event
2021-05-03 23:33:03.220 [debug] <0.294.0> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event
2021-05-03 23:33:03.220 [debug] <0.297.0> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event
2021-05-03 23:33:03.223 [debug] <0.300.0> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event
2021-05-03 23:33:03.234 [debug] <0.303.0> Lager installed handler lager_forwarder_backend into rabbit_log_feature_flags_lager_event
2021-05-03 23:33:03.258 [debug] <0.306.0> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event
2021-05-03 23:33:03.274 [debug] <0.309.0> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event
2021-05-03 23:33:03.326 [debug] <0.312.0> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event
2021-05-03 23:33:03.351 [debug] <0.315.0> Lager installed handler lager_forwarder_backend into rabbit_log_prelaunch_lager_event
2021-05-03 23:33:03.390 [debug] <0.318.0> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event
2021-05-03 23:33:03.392 [debug] <0.321.0> Lager installed handler lager_forwarder_backend into rabbit_log_ra_lager_event
2021-05-03 23:33:03.402 [debug] <0.324.0> Lager installed handler lager_forwarder_backend into rabbit_log_shovel_lager_event
2021-05-03 23:33:03.426 [debug] <0.327.0> Lager installed handler lager_forwarder_backend into rabbit_log_upgrade_lager_event
2021-05-03 23:33:03.494 [info] <0.44.0> Application lager started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:03.666 [debug] <0.284.0> Lager installed handler lager_backend_throttle into lager_event
2021-05-03 23:33:07.460 [info] <0.44.0> Application mnesia started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:07.461 [info] <0.273.0>
Starting RabbitMQ 3.8.14 on Erlang 23.3.2
Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
Licensed under the MPL 2.0. Website: https://rabbitmq.com
## ## RabbitMQ 3.8.14
## ##
########## Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
###### ##
########## Licensed under the MPL 2.0. Website: https://rabbitmq.com
Doc guides: https://rabbitmq.com/documentation.html
Support: https://rabbitmq.com/contact.html
Tutorials: https://rabbitmq.com/getstarted.html
Monitoring: https://rabbitmq.com/monitoring.html
Logs:
Config file(s): /etc/rabbitmq/rabbitmq.conf
/etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf
Starting broker...2021-05-03 23:33:07.463 [info] <0.273.0>
node : rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.conf
: /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf
cookie hash : uNeJA1o1sMgjKEUf2Th5yw==
log(s) :
database dir : /var/lib/rabbitmq/mnesia/rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt
2021-05-03 23:33:08.507 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:08.507 [info] <0.273.0> Feature flags: [ ] drop_unroutable_metric
2021-05-03 23:33:08.507 [info] <0.273.0> Feature flags: [ ] empty_basic_get_metric
2021-05-03 23:33:08.507 [info] <0.273.0> Feature flags: [ ] implicit_default_bindings
2021-05-03 23:33:08.507 [info] <0.273.0> Feature flags: [ ] maintenance_mode_status
2021-05-03 23:33:08.507 [info] <0.273.0> Feature flags: [ ] quorum_queue
2021-05-03 23:33:08.507 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:08.507 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:08.507 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:18.207 [info] <0.273.0> Running boot step pre_boot defined by app rabbit
2021-05-03 23:33:18.207 [info] <0.273.0> Running boot step rabbit_core_metrics defined by app rabbit
2021-05-03 23:33:18.208 [info] <0.273.0> Running boot step rabbit_alarm defined by app rabbit
2021-05-03 23:33:18.347 [info] <0.358.0> Memory high watermark set to 6415 MiB (6727421132 bytes) of 16039 MiB (16818552832 bytes) total
2021-05-03 23:33:18.363 [info] <0.360.0> Enabling free disk space monitoring
2021-05-03 23:33:18.363 [info] <0.360.0> Disk free limit set to 50MB
2021-05-03 23:33:18.377 [info] <0.273.0> Running boot step code_server_cache defined by app rabbit
2021-05-03 23:33:18.378 [info] <0.273.0> Running boot step file_handle_cache defined by app rabbit
2021-05-03 23:33:18.379 [info] <0.364.0> FHC read buffering: OFF
2021-05-03 23:33:18.379 [info] <0.364.0> FHC write buffering: ON
2021-05-03 23:33:18.379 [info] <0.363.0> Limiting to approx 1048479 file handles (943629 sockets)
2021-05-03 23:33:18.396 [info] <0.273.0> Running boot step worker_pool defined by app rabbit
2021-05-03 23:33:18.396 [info] <0.348.0> Will use 8 processes for default worker pool
2021-05-03 23:33:18.397 [info] <0.348.0> Starting worker pool 'worker_pool' with 8 processes in it
2021-05-03 23:33:18.398 [info] <0.273.0> Running boot step database defined by app rabbit
2021-05-03 23:33:18.398 [info] <0.273.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt is empty. Assuming we need to join an existing cluster or initialise from scratch...
2021-05-03 23:33:18.398 [info] <0.273.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2021-05-03 23:33:18.398 [info] <0.273.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2021-05-03 23:33:18.399 [info] <0.273.0> Peer discovery backend does not support locking, falling back to randomized delay
2021-05-03 23:33:18.399 [info] <0.273.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2021-05-03 23:33:18.399 [info] <0.273.0> All discovered existing cluster peers:
2021-05-03 23:33:18.399 [info] <0.273.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
2021-05-03 23:33:18.405 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-05-03 23:33:18.405 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-05-03 23:33:19.963 [info] <0.44.0> Application mnesia started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:24.744 [info] <0.273.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-03 23:33:24.744 [info] <0.273.0> Successfully synced tables from a peer
2021-05-03 23:33:24.813 [info] <0.273.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-03 23:33:24.813 [info] <0.273.0> Successfully synced tables from a peer
2021-05-03 23:33:24.813 [info] <0.273.0> Feature flag `drop_unroutable_metric`: supported, attempt to enable...
2021-05-03 23:33:24.813 [info] <0.273.0> Feature flag `drop_unroutable_metric`: mark as enabled=state_changing
2021-05-03 23:33:24.824 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:24.824 [info] <0.273.0> Feature flags: [~] drop_unroutable_metric
2021-05-03 23:33:24.824 [info] <0.273.0> Feature flags: [ ] empty_basic_get_metric
2021-05-03 23:33:24.824 [info] <0.273.0> Feature flags: [ ] implicit_default_bindings
2021-05-03 23:33:24.824 [info] <0.273.0> Feature flags: [ ] maintenance_mode_status
2021-05-03 23:33:24.824 [info] <0.273.0> Feature flags: [ ] quorum_queue
2021-05-03 23:33:24.825 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:24.825 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:24.825 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:24.842 [info] <0.273.0> Feature flag `drop_unroutable_metric`: mark as enabled=true
2021-05-03 23:33:24.866 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:24.866 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:24.866 [info] <0.273.0> Feature flags: [ ] empty_basic_get_metric
2021-05-03 23:33:24.867 [info] <0.273.0> Feature flags: [ ] implicit_default_bindings
2021-05-03 23:33:24.867 [info] <0.273.0> Feature flags: [ ] maintenance_mode_status
2021-05-03 23:33:24.867 [info] <0.273.0> Feature flags: [ ] quorum_queue
2021-05-03 23:33:24.868 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:24.868 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:24.869 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:24.912 [info] <0.273.0> Feature flag `empty_basic_get_metric`: supported, attempt to enable...
2021-05-03 23:33:24.913 [info] <0.273.0> Feature flag `empty_basic_get_metric`: mark as enabled=state_changing
2021-05-03 23:33:24.922 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:24.922 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:24.922 [info] <0.273.0> Feature flags: [~] empty_basic_get_metric
2021-05-03 23:33:24.922 [info] <0.273.0> Feature flags: [ ] implicit_default_bindings
2021-05-03 23:33:24.923 [info] <0.273.0> Feature flags: [ ] maintenance_mode_status
2021-05-03 23:33:24.923 [info] <0.273.0> Feature flags: [ ] quorum_queue
2021-05-03 23:33:24.923 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:24.923 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:24.923 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:24.939 [info] <0.273.0> Feature flag `empty_basic_get_metric`: mark as enabled=true
2021-05-03 23:33:25.017 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:25.017 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:25.017 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:25.017 [info] <0.273.0> Feature flags: [ ] implicit_default_bindings
2021-05-03 23:33:25.018 [info] <0.273.0> Feature flags: [ ] maintenance_mode_status
2021-05-03 23:33:25.018 [info] <0.273.0> Feature flags: [ ] quorum_queue
2021-05-03 23:33:25.018 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:25.018 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:25.018 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:25.042 [info] <0.273.0> Feature flag `implicit_default_bindings`: supported, attempt to enable...
2021-05-03 23:33:25.043 [info] <0.273.0> Feature flag `implicit_default_bindings`: mark as enabled=state_changing
2021-05-03 23:33:25.057 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:25.058 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:25.058 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:25.058 [info] <0.273.0> Feature flags: [~] implicit_default_bindings
2021-05-03 23:33:25.058 [info] <0.273.0> Feature flags: [ ] maintenance_mode_status
2021-05-03 23:33:25.058 [info] <0.273.0> Feature flags: [ ] quorum_queue
2021-05-03 23:33:25.058 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:25.059 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:25.059 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:25.112 [info] <0.273.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2021-05-03 23:33:25.112 [info] <0.273.0> Successfully synced tables from a peer
2021-05-03 23:33:25.112 [info] <0.273.0> Feature flag `implicit_default_bindings`: mark as enabled=true
2021-05-03 23:33:25.125 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:25.125 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:25.125 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:25.125 [info] <0.273.0> Feature flags: [x] implicit_default_bindings
2021-05-03 23:33:25.125 [info] <0.273.0> Feature flags: [ ] maintenance_mode_status
2021-05-03 23:33:25.125 [info] <0.273.0> Feature flags: [ ] quorum_queue
2021-05-03 23:33:25.125 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:25.125 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:25.126 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:25.173 [info] <0.273.0> Feature flag `maintenance_mode_status`: supported, attempt to enable...
2021-05-03 23:33:25.173 [info] <0.273.0> Feature flag `maintenance_mode_status`: mark as enabled=state_changing
2021-05-03 23:33:25.185 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:25.185 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:25.185 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:25.185 [info] <0.273.0> Feature flags: [x] implicit_default_bindings
2021-05-03 23:33:25.185 [info] <0.273.0> Feature flags: [~] maintenance_mode_status
2021-05-03 23:33:25.185 [info] <0.273.0> Feature flags: [ ] quorum_queue
2021-05-03 23:33:25.185 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:25.185 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:25.185 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:25.206 [info] <0.273.0> Creating table rabbit_node_maintenance_states for feature flag `maintenance_mode_status`
2021-05-03 23:33:25.247 [info] <0.273.0> Feature flag `maintenance_mode_status`: mark as enabled=true
2021-05-03 23:33:25.270 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:25.270 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:25.270 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:25.270 [info] <0.273.0> Feature flags: [x] implicit_default_bindings
2021-05-03 23:33:25.270 [info] <0.273.0> Feature flags: [x] maintenance_mode_status
2021-05-03 23:33:25.270 [info] <0.273.0> Feature flags: [ ] quorum_queue
2021-05-03 23:33:25.270 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:25.270 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:25.270 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:25.302 [info] <0.273.0> Feature flag `quorum_queue`: supported, attempt to enable...
2021-05-03 23:33:25.302 [info] <0.273.0> Feature flag `quorum_queue`: mark as enabled=state_changing
2021-05-03 23:33:25.315 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:25.316 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:25.316 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:25.316 [info] <0.273.0> Feature flags: [x] implicit_default_bindings
2021-05-03 23:33:25.316 [info] <0.273.0> Feature flags: [x] maintenance_mode_status
2021-05-03 23:33:25.316 [info] <0.273.0> Feature flags: [~] quorum_queue
2021-05-03 23:33:25.316 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:25.316 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:25.316 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:25.331 [info] <0.273.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-03 23:33:25.331 [info] <0.273.0> Successfully synced tables from a peer
2021-05-03 23:33:25.332 [info] <0.273.0> Feature flag `quorum_queue`: migrating Mnesia table rabbit_queue...
2021-05-03 23:33:25.540 [info] <0.273.0> Feature flag `quorum_queue`: migrating Mnesia table rabbit_durable_queue...
2021-05-03 23:33:26.011 [info] <0.273.0> Feature flag `quorum_queue`: Mnesia tables migration done
2021-05-03 23:33:26.011 [info] <0.273.0> Feature flag `quorum_queue`: mark as enabled=true
2021-05-03 23:33:26.022 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:26.022 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:26.022 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:26.022 [info] <0.273.0> Feature flags: [x] implicit_default_bindings
2021-05-03 23:33:26.022 [info] <0.273.0> Feature flags: [x] maintenance_mode_status
2021-05-03 23:33:26.022 [info] <0.273.0> Feature flags: [x] quorum_queue
2021-05-03 23:33:26.022 [info] <0.273.0> Feature flags: [ ] user_limits
2021-05-03 23:33:26.022 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:26.022 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:26.054 [info] <0.273.0> Feature flag `user_limits`: supported, attempt to enable...
2021-05-03 23:33:26.055 [info] <0.273.0> Feature flag `user_limits`: mark as enabled=state_changing
2021-05-03 23:33:26.069 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:26.070 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:26.070 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:26.070 [info] <0.273.0> Feature flags: [x] implicit_default_bindings
2021-05-03 23:33:26.070 [info] <0.273.0> Feature flags: [x] maintenance_mode_status
2021-05-03 23:33:26.070 [info] <0.273.0> Feature flags: [x] quorum_queue
2021-05-03 23:33:26.070 [info] <0.273.0> Feature flags: [~] user_limits
2021-05-03 23:33:26.070 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:26.070 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:26.090 [info] <0.273.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-03 23:33:26.090 [info] <0.273.0> Successfully synced tables from a peer
2021-05-03 23:33:26.302 [info] <0.273.0> Feature flag `user_limits`: mark as enabled=true
2021-05-03 23:33:26.328 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:26.328 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:26.328 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:26.328 [info] <0.273.0> Feature flags: [x] implicit_default_bindings
2021-05-03 23:33:26.328 [info] <0.273.0> Feature flags: [x] maintenance_mode_status
2021-05-03 23:33:26.328 [info] <0.273.0> Feature flags: [x] quorum_queue
2021-05-03 23:33:26.328 [info] <0.273.0> Feature flags: [x] user_limits
2021-05-03 23:33:26.328 [info] <0.273.0> Feature flags: [ ] virtual_host_metadata
2021-05-03 23:33:26.328 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:26.342 [info] <0.273.0> Feature flag `virtual_host_metadata`: supported, attempt to enable...
2021-05-03 23:33:26.342 [info] <0.273.0> Feature flag `virtual_host_metadata`: mark as enabled=state_changing
2021-05-03 23:33:26.357 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:26.357 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:26.357 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:26.358 [info] <0.273.0> Feature flags: [x] implicit_default_bindings
2021-05-03 23:33:26.358 [info] <0.273.0> Feature flags: [x] maintenance_mode_status
2021-05-03 23:33:26.358 [info] <0.273.0> Feature flags: [x] quorum_queue
2021-05-03 23:33:26.358 [info] <0.273.0> Feature flags: [x] user_limits
2021-05-03 23:33:26.358 [info] <0.273.0> Feature flags: [~] virtual_host_metadata
2021-05-03 23:33:26.358 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:26.390 [info] <0.273.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-03 23:33:26.395 [info] <0.273.0> Successfully synced tables from a peer
2021-05-03 23:33:26.607 [info] <0.273.0> Feature flag `virtual_host_metadata`: mark as enabled=true
2021-05-03 23:33:26.647 [info] <0.273.0> Feature flags: list of feature flags found:
2021-05-03 23:33:26.647 [info] <0.273.0> Feature flags: [x] drop_unroutable_metric
2021-05-03 23:33:26.647 [info] <0.273.0> Feature flags: [x] empty_basic_get_metric
2021-05-03 23:33:26.647 [info] <0.273.0> Feature flags: [x] implicit_default_bindings
2021-05-03 23:33:26.647 [info] <0.273.0> Feature flags: [x] maintenance_mode_status
2021-05-03 23:33:26.647 [info] <0.273.0> Feature flags: [x] quorum_queue
2021-05-03 23:33:26.647 [info] <0.273.0> Feature flags: [x] user_limits
2021-05-03 23:33:26.647 [info] <0.273.0> Feature flags: [x] virtual_host_metadata
2021-05-03 23:33:26.647 [info] <0.273.0> Feature flags: feature flag states written to disk: yes
2021-05-03 23:33:26.696 [info] <0.273.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-03 23:33:26.696 [info] <0.273.0> Successfully synced tables from a peer
2021-05-03 23:33:26.953 [info] <0.273.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-03 23:33:26.953 [info] <0.273.0> Successfully synced tables from a peer
2021-05-03 23:33:26.953 [info] <0.273.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2021-05-03 23:33:26.953 [info] <0.273.0> Running boot step database_sync defined by app rabbit
2021-05-03 23:33:26.954 [info] <0.273.0> Running boot step feature_flags defined by app rabbit
2021-05-03 23:33:26.954 [info] <0.273.0> Running boot step codec_correctness_check defined by app rabbit
2021-05-03 23:33:26.954 [info] <0.273.0> Running boot step external_infrastructure defined by app rabbit
2021-05-03 23:33:26.954 [info] <0.273.0> Running boot step rabbit_registry defined by app rabbit
2021-05-03 23:33:26.954 [info] <0.273.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-05-03 23:33:26.954 [info] <0.273.0> Running boot step rabbit_queue_location_random defined by app rabbit
2021-05-03 23:33:26.954 [info] <0.273.0> Running boot step rabbit_event defined by app rabbit
2021-05-03 23:33:26.955 [info] <0.273.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-05-03 23:33:26.955 [info] <0.273.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-05-03 23:33:26.955 [info] <0.273.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-05-03 23:33:26.955 [info] <0.273.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-05-03 23:33:26.955 [info] <0.273.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-05-03 23:33:26.955 [info] <0.273.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-05-03 23:33:26.955 [info] <0.273.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-05-03 23:33:26.956 [info] <0.273.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-05-03 23:33:26.956 [info] <0.273.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-05-03 23:33:26.956 [info] <0.273.0> Running boot step rabbit_priority_queue defined by app rabbit
2021-05-03 23:33:26.956 [info] <0.273.0> Priority queues enabled, real BQ is rabbit_variable_queue
2021-05-03 23:33:26.956 [info] <0.273.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-05-03 23:33:26.956 [info] <0.273.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-05-03 23:33:26.956 [info] <0.273.0> Running boot step kernel_ready defined by app rabbit
2021-05-03 23:33:26.956 [info] <0.273.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-05-03 23:33:26.956 [info] <0.273.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-05-03 23:33:26.970 [info] <0.628.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-05-03 23:33:26.970 [info] <0.273.0> Running boot step guid_generator defined by app rabbit
2021-05-03 23:33:27.193 [info] <0.273.0> Running boot step rabbit_node_monitor defined by app rabbit
2021-05-03 23:33:27.194 [info] <0.632.0> Starting rabbit_node_monitor
2021-05-03 23:33:27.194 [info] <0.273.0> Running boot step delegate_sup defined by app rabbit
2021-05-03 23:33:27.195 [info] <0.273.0> Running boot step rabbit_memory_monitor defined by app rabbit
2021-05-03 23:33:27.196 [info] <0.273.0> Running boot step core_initialized defined by app rabbit
2021-05-03 23:33:27.196 [info] <0.273.0> Running boot step upgrade_queues defined by app rabbit
2021-05-03 23:33:27.292 [info] <0.273.0> message_store upgrades: 1 to apply
2021-05-03 23:33:27.292 [info] <0.273.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2021-05-03 23:33:27.293 [info] <0.273.0> message_store upgrades: No durable queues found. Skipping message store migration
2021-05-03 23:33:27.293 [info] <0.273.0> message_store upgrades: Removing the old message store data
2021-05-03 23:33:27.301 [info] <0.273.0> message_store upgrades: All upgrades applied successfully
2021-05-03 23:33:27.419 [info] <0.273.0> Running boot step channel_tracking defined by app rabbit
2021-05-03 23:33:27.492 [info] <0.273.0> Setting up a table for channel tracking on this node: 'tracked_channel_on_node_rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.520 [info] <0.273.0> Setting up a table for channel tracking on this node: 'tracked_channel_table_per_user_on_node_rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.521 [info] <0.273.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2021-05-03 23:33:27.521 [info] <0.273.0> Running boot step connection_tracking defined by app rabbit
2021-05-03 23:33:27.567 [info] <0.273.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.596 [info] <0.273.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.623 [info] <0.273.0> Setting up a table for per-user connection counting on this node: 'tracked_connection_table_per_user_on_node_rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.624 [info] <0.273.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-05-03 23:33:27.624 [info] <0.273.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-05-03 23:33:27.624 [info] <0.273.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-05-03 23:33:27.624 [info] <0.273.0> Running boot step rabbit_policies defined by app rabbit
2021-05-03 23:33:27.626 [info] <0.273.0> Running boot step rabbit_policy defined by app rabbit
2021-05-03 23:33:27.626 [info] <0.273.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-05-03 23:33:27.626 [info] <0.273.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-05-03 23:33:27.626 [info] <0.273.0> Running boot step rabbit_vhost_limit defined by app rabbit
2021-05-03 23:33:27.626 [info] <0.273.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2021-05-03 23:33:27.627 [info] <0.273.0> Management plugin: using rates mode 'basic'
2021-05-03 23:33:27.627 [info] <0.273.0> Running boot step recovery defined by app rabbit
2021-05-03 23:33:27.628 [info] <0.273.0> Running boot step empty_db_check defined by app rabbit
2021-05-03 23:33:27.628 [info] <0.273.0> Will seed default virtual host and user...
2021-05-03 23:33:27.629 [info] <0.273.0> Adding vhost '/' (description: 'Default virtual host')
2021-05-03 23:33:27.730 [info] <0.691.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-05-03 23:33:27.750 [info] <0.691.0> Starting message stores for vhost '/'
2021-05-03 23:33:27.751 [info] <0.695.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-05-03 23:33:27.757 [info] <0.691.0> Started message store of type transient for vhost '/'
2021-05-03 23:33:27.757 [info] <0.699.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-05-03 23:33:27.769 [warning] <0.699.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2021-05-03 23:33:27.793 [info] <0.691.0> Started message store of type persistent for vhost '/'
2021-05-03 23:33:27.856 [info] <0.273.0> Created user 'guest'
2021-05-03 23:33:27.860 [info] <0.273.0> Successfully set user tags for user 'guest' to [administrator]
2021-05-03 23:33:27.867 [info] <0.273.0> Successfully set permissions for 'guest' in virtual host '/' to '.*', '.*', '.*'
2021-05-03 23:33:27.867 [info] <0.273.0> Running boot step rabbit_looking_glass defined by app rabbit
2021-05-03 23:33:27.867 [info] <0.273.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-05-03 23:33:27.896 [info] <0.273.0> Running boot step background_gc defined by app rabbit
2021-05-03 23:33:27.897 [info] <0.273.0> Running boot step routing_ready defined by app rabbit
2021-05-03 23:33:27.897 [info] <0.273.0> Running boot step pre_flight defined by app rabbit
2021-05-03 23:33:27.897 [info] <0.273.0> Running boot step notify_cluster defined by app rabbit
2021-05-03 23:33:27.897 [info] <0.273.0> Running boot step networking defined by app rabbit
2021-05-03 23:33:27.897 [info] <0.273.0> Running boot step definition_import_worker_pool defined by app rabbit
2021-05-03 23:33:27.897 [info] <0.348.0> Starting worker pool 'definition_import_pool' with 8 processes in it
2021-05-03 23:33:27.907 [info] <0.273.0> Running boot step cluster_name defined by app rabbit
2021-05-03 23:33:27.907 [info] <0.273.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-3WCHXjCVOX9Yc9ZVTpC3Mg'
2021-05-03 23:33:27.946 [info] <0.273.0> Running boot step direct_client defined by app rabbit
2021-05-03 23:33:27.946 [info] <0.736.0> Resetting node maintenance status
2021-05-03 23:33:27.946 [info] <0.44.0> Application rabbit started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.946 [info] <0.44.0> Application cowlib started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.947 [info] <0.44.0> Application cowboy started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.974 [info] <0.44.0> Application rabbitmq_management_agent started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.996 [info] <0.44.0> Application prometheus started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:27.996 [info] <0.44.0> Application rabbitmq_web_dispatch started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:28.044 [info] <0.761.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2021-05-03 23:33:28.044 [info] <0.44.0> Application rabbitmq_prometheus started on node 'rabbit@dev-multicloud-fcaps-7cdbcbf95f-cb4tt'
2021-05-03 23:33:28.044 [info] <0.736.0> Ready to start client connection listeners
2021-05-03 23:33:28.062 [info] <0.880.0> started TCP listener on [::]:5672
completed with 3 plugins.
2021-05-03 23:33:28.933 [info] <0.736.0> Server startup complete; 3 plugins started.
* rabbitmq_prometheus
* rabbitmq_web_dispatch
* rabbitmq_management_agent
2021-05-03 23:33:28.933 [info] <0.736.0> Resetting node maintenance status
2021-05-03 23:33:40.977 [info] <0.886.0> accepting AMQP connection <0.886.0> (127.0.0.1:47502 -> 127.0.0.1:5672)
2021-05-03 23:33:41.174 [info] <0.886.0> connection <0.886.0> (127.0.0.1:47502 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-05-03 23:33:41.208 [info] <0.895.0> accepting AMQP connection <0.895.0> (127.0.0.1:47506 -> 127.0.0.1:5672)
2021-05-03 23:33:41.231 [info] <0.895.0> connection <0.895.0> (127.0.0.1:47506 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-05-03 23:33:41.267 [info] <0.912.0> accepting AMQP connection <0.912.0> (127.0.0.1:47508 -> 127.0.0.1:5672)
2021-05-03 23:33:41.274 [info] <0.912.0> connection <0.912.0> (127.0.0.1:47508 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
{removed_failing_handler,rabbit_log}
2021-05-04 00:52:40.613 [error] <0.239.0> ** Generic server aten_detector terminating
** Last message in was poll
** When Server state == {state,#Ref<0.4265700356.2991849473.38480>,5000,0.99,#{},#{}}
** Reason for termination ==
** {{timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}},[{gen_server,call,2,[{file,"gen_server.erl"},{line,238}]},{aten_detector,handle_info,2,[{file,"src/aten_detector.erl"},{line,103}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2021-05-04 00:52:40.614 [error] <0.239.0> CRASH REPORT Process aten_detector with 0 neighbours exited with reason: {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in gen_server:call/2 line 238
2021-05-04 00:52:40.614 [error] <0.236.0> Supervisor aten_sup had child aten_detector started with aten_detector:start_link() at <0.239.0> exit with reason {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in context child_terminated
2021-05-04 01:10:50.689 [error] <0.886.0> closing AMQP connection <0.886.0> (127.0.0.1:47502 -> 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s
2021-05-04 01:10:51.091 [info] <0.2035.0> Closing all channels from connection '127.0.0.1:47502 -> 127.0.0.1:5672' because it has been closed
2021-05-04 01:10:51.423 [error] <0.899.0> Supervisor {<0.899.0>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.886.0>, <0.900.0>, <0.886.0>, <<"127.0.0.1:47502 -> 127.0.0.1:5672">>, rabbit_framing_amqp_0_9_1, {user,<<"guest">>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"consumer_cancel_notify">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"authentication...">>,...}], <0.888.0>, <0.901.0>) at <0.902.0> exit with reason noproc in context shutdown_error
2021-05-04 01:10:54.557 [info] <0.2046.0> accepting AMQP connection <0.2046.0> (127.0.0.1:32848 -> 127.0.0.1:5672)
2021-05-04 01:10:54.805 [info] <0.2046.0> connection <0.2046.0> (127.0.0.1:32848 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-05-04 01:10:54.853 [info] <0.895.0> closing AMQP connection <0.895.0> (127.0.0.1:47506 -> 127.0.0.1:5672, vhost: '/', user: 'guest')
2021-05-04 01:10:54.854 [info] <0.2055.0> Closing all channels from connection '127.0.0.1:47506 -> 127.0.0.1:5672' because it has been closed
2021-05-04 01:10:54.869 [info] <0.2058.0> accepting AMQP connection <0.2058.0> (127.0.0.1:32872 -> 127.0.0.1:5672)
2021-05-04 01:10:54.879 [info] <0.2058.0> connection <0.2058.0> (127.0.0.1:32872 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-05-04 01:29:48.713 [error] <0.1830.0> ** Generic server aten_detector terminating
** Last message in was poll
** When Server state == {state,#Ref<0.4265700356.2991849473.61579>,5000,0.99,#{},#{}}
** Reason for termination ==
** {{timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}},[{gen_server,call,2,[{file,"gen_server.erl"},{line,238}]},{aten_detector,handle_info,2,[{file,"src/aten_detector.erl"},{line,103}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2021-05-04 01:29:48.714 [error] <0.1830.0> CRASH REPORT Process aten_detector with 0 neighbours exited with reason: {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in gen_server:call/2 line 238
2021-05-04 01:29:48.714 [error] <0.236.0> Supervisor aten_sup had child aten_detector started with aten_detector:start_link() at <0.1830.0> exit with reason {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in context child_terminated
2021-05-04 01:39:57.101 [error] <0.2046.0> closing AMQP connection <0.2046.0> (127.0.0.1:32848 -> 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s
2021-05-04 01:39:57.131 [info] <0.2406.0> Closing all channels from connection '127.0.0.1:32848 -> 127.0.0.1:5672' because it has been closed
2021-05-04 01:39:59.217 [info] <0.2418.0> accepting AMQP connection <0.2418.0> (127.0.0.1:37184 -> 127.0.0.1:5672)
2021-05-04 01:39:59.588 [info] <0.2418.0> connection <0.2418.0> (127.0.0.1:37184 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-05-04 01:39:59.598 [info] <0.2058.0> closing AMQP connection <0.2058.0> (127.0.0.1:32872 -> 127.0.0.1:5672, vhost: '/', user: 'guest')
2021-05-04 01:39:59.602 [info] <0.2427.0> Closing all channels from connection '127.0.0.1:32872 -> 127.0.0.1:5672' because it has been closed
2021-05-04 01:39:59.675 [info] <0.2430.0> accepting AMQP connection <0.2430.0> (127.0.0.1:37194 -> 127.0.0.1:5672)
2021-05-04 01:39:59.799 [info] <0.2430.0> connection <0.2430.0> (127.0.0.1:37194 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'