OpenDistro for Elasticsearch Security Demo Installer ** Warning: Do not use on production or public reachable systems ** Basedir: /usr/share/elasticsearch Elasticsearch install type: rpm/deb on CentOS Linux release 7.7.1908 (Core) Elasticsearch config dir: /usr/share/elasticsearch/config Elasticsearch config file: /usr/share/elasticsearch/config/elasticsearch.yml Elasticsearch bin dir: /usr/share/elasticsearch/bin Elasticsearch plugins dir: /usr/share/elasticsearch/plugins Elasticsearch lib dir: /usr/share/elasticsearch/lib Detected Elasticsearch Version: x-content-7.3.2 Detected Open Distro Security Version: 1.3.0.0 /usr/share/elasticsearch/config/elasticsearch.yml seems to be already configured for Security. Quit. OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2021-04-14T02:12:43,981][WARN ][o.e.d.c.s.Settings ] [cldash-es-node1] [transport.tcp.port] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version. [2021-04-14T02:12:44,343][WARN ][o.e.d.c.s.Settings ] [cldash-es-node1] [node.max_local_storage_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version. [2021-04-14T02:12:44,661][INFO ][o.e.e.NodeEnvironment ] [cldash-es-node1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (10.253.0.188:/dockerdata-nfs/onap/clamp/dashboard-elasticsearch/data)]], net usable_space [94.1gb], net total_space [99.9gb], types [nfs4] [2021-04-14T02:12:44,661][INFO ][o.e.e.NodeEnvironment ] [cldash-es-node1] heap size [1015.6mb], compressed ordinary object pointers [true] [2021-04-14T02:12:44,675][INFO ][o.e.n.Node ] [cldash-es-node1] node name [cldash-es-node1], node ID [THYoUU8dSEycjJ_09nvebw], cluster name [clamp-dashboard] [2021-04-14T02:12:44,675][INFO ][o.e.n.Node ] [cldash-es-node1] version[7.3.2], pid[1], build[oss/tar/1c1faf1/2019-09-06T14:40:30.409026Z], OS[Linux/4.9.0-14-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13] [2021-04-14T02:12:44,675][INFO ][o.e.n.Node ] [cldash-es-node1] JVM home [/opt/jdk-11.0.1] [2021-04-14T02:12:44,676][INFO ][o.e.n.Node ] [cldash-es-node1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-12708674415328466886, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Djava.security.policy=file:///usr/share/elasticsearch/plugins/opendistro_performance_analyzer/pa_config/es_security.policy, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Des.cgroups.hierarchy.override=/, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true] [2021-04-14T02:12:47,862][INFO ][c.a.o.e.p.c.PluginSettings] [cldash-es-node1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true [2021-04-14T02:12:48,444][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] ES Config path is /usr/share/elasticsearch/config [2021-04-14T02:12:48,575][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [cldash-es-node1] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL [2021-04-14T02:12:48,968][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [cldash-es-node1] JVM supports TLSv1.3 [2021-04-14T02:12:48,969][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [cldash-es-node1] Config directory is /usr/share/elasticsearch/config/, from there the key- and truststore files are resolved relatively [2021-04-14T02:12:50,656][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [cldash-es-node1] TLS Transport Client Provider : JDK [2021-04-14T02:12:50,656][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [cldash-es-node1] TLS Transport Server Provider : JDK [2021-04-14T02:12:50,656][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [cldash-es-node1] TLS HTTP Provider : JDK [2021-04-14T02:12:50,656][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [cldash-es-node1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1] [2021-04-14T02:12:50,657][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [cldash-es-node1] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2, TLSv1.1] [2021-04-14T02:12:51,656][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] Clustername: clamp-dashboard [2021-04-14T02:12:51,761][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] Directory /usr/share/elasticsearch/config has insecure file permissions (should be 0700) [2021-04-14T02:12:51,762][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/elasticsearch.yml has insecure file permissions (should be 0600) [2021-04-14T02:12:51,762][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/log4j2.properties has insecure file permissions (should be 0600) [2021-04-14T02:12:51,762][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/org.onap.clamp.crt.key has insecure file permissions (should be 0600) [2021-04-14T02:12:51,762][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/org.onap.clamp.key.pem has insecure file permissions (should be 0600) [2021-04-14T02:12:51,763][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/clamp-ca-certs.pem has insecure file permissions (should be 0600) [2021-04-14T02:12:51,763][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/kirk-key.pem has insecure file permissions (should be 0600) [2021-04-14T02:12:51,763][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/kirk.pem has insecure file permissions (should be 0600) [2021-04-14T02:12:51,763][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/esnode-key.pem has insecure file permissions (should be 0600) [2021-04-14T02:12:51,763][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/esnode.pem has insecure file permissions (should be 0600) [2021-04-14T02:12:51,764][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] File /usr/share/elasticsearch/config/root-ca.pem has insecure file permissions (should be 0600) [2021-04-14T02:12:52,345][INFO ][c.a.o.j.JobSchedulerPlugin] [cldash-es-node1] Loaded scheduler extension: opendistro-managed-index, index: .opendistro-ism-config [2021-04-14T02:12:52,468][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [aggs-matrix-stats] [2021-04-14T02:12:52,468][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [analysis-common] [2021-04-14T02:12:52,468][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [ingest-common] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [ingest-geoip] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [ingest-user-agent] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [lang-expression] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [lang-mustache] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [lang-painless] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [mapper-extras] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [parent-join] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [percolator] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [rank-eval] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [reindex] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [repository-url] [2021-04-14T02:12:52,469][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded module [transport-netty4] [2021-04-14T02:12:52,470][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded plugin [opendistro-job-scheduler] [2021-04-14T02:12:52,470][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded plugin [opendistro_alerting] [2021-04-14T02:12:52,470][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded plugin [opendistro_index_management] [2021-04-14T02:12:52,470][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded plugin [opendistro_performance_analyzer] [2021-04-14T02:12:52,470][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded plugin [opendistro_security] [2021-04-14T02:12:52,470][INFO ][o.e.p.PluginsService ] [cldash-es-node1] loaded plugin [opendistro_sql] [2021-04-14T02:12:52,486][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml [2021-04-14T02:12:59,184][WARN ][o.e.d.c.s.Settings ] [cldash-es-node1] [discovery.zen.minimum_master_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version. [2021-04-14T02:13:01,484][INFO ][c.a.o.s.a.i.AuditLogImpl ] [cldash-es-node1] Configured categories on rest layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2021-04-14T02:13:01,485][INFO ][c.a.o.s.a.i.AuditLogImpl ] [cldash-es-node1] Configured categories on transport layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2021-04-14T02:13:01,485][INFO ][c.a.o.s.a.i.AuditLogImpl ] [cldash-es-node1] Configured Users to ignore: [kibanaserver] [2021-04-14T02:13:01,485][INFO ][c.a.o.s.a.i.AuditLogImpl ] [cldash-es-node1] Configured Users to ignore for read compliance events: [kibanaserver] [2021-04-14T02:13:01,485][INFO ][c.a.o.s.a.i.AuditLogImpl ] [cldash-es-node1] Configured Users to ignore for write compliance events: [kibanaserver] [2021-04-14T02:13:01,887][INFO ][c.a.o.s.a.i.AuditLogImpl ] [cldash-es-node1] Message routing enabled: true [2021-04-14T02:13:01,951][WARN ][c.a.o.s.c.ComplianceConfig] [cldash-es-node1] If you plan to use field masking pls configure opendistro_security.compliance.salt to be a random string of 16 chars length identical on all nodes [2021-04-14T02:13:01,951][INFO ][c.a.o.s.c.ComplianceConfig] [cldash-es-node1] PII configuration [auditLogPattern=org.joda.time.format.DateTimeFormatter@de0c402, auditLogIndex=null]: {} [2021-04-14T02:13:03,352][INFO ][o.e.d.DiscoveryModule ] [cldash-es-node1] using discovery type [zen] and seed hosts providers [settings] [2021-04-14T02:13:04,675][INFO ][c.a.o.e.p.h.c.PerformanceAnalyzerConfigAction] [cldash-es-node1] PerformanceAnalyzer Enabled: true [2021-04-14T02:13:04,844][INFO ][o.e.n.Node ] [cldash-es-node1] initialized [2021-04-14T02:13:04,845][INFO ][o.e.n.Node ] [cldash-es-node1] starting ... [2021-04-14T02:13:05,768][INFO ][o.e.t.TransportService ] [cldash-es-node1] publish_address {10.233.72.1:9300}, bound_addresses {0.0.0.0:9300} [2021-04-14T02:13:05,862][INFO ][o.e.b.BootstrapChecks ] [cldash-es-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks [2021-04-14T02:13:06,065][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [cldash-es-node1] Collector HeapMetrics is still in progress, so skipping this Interval [2021-04-14T02:13:06,068][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [cldash-es-node1] Collector NetworkInterfaceCollector is still in progress, so skipping this Interval [2021-04-14T02:13:06,068][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [cldash-es-node1] Collector ThreadPoolMetrics is still in progress, so skipping this Interval [2021-04-14T02:13:06,068][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [cldash-es-node1] Collector CircuitBreaker is still in progress, so skipping this Interval [2021-04-14T02:13:06,148][INFO ][o.e.c.c.Coordinator ] [cldash-es-node1] setting initial configuration to VotingConfiguration{THYoUU8dSEycjJ_09nvebw} [2021-04-14T02:13:07,252][INFO ][o.e.c.c.JoinHelper ] [cldash-es-node1] failed to join {cldash-es-node1}{THYoUU8dSEycjJ_09nvebw}{7Ui6jtwVS7OPTOJXUsergQ}{10.233.72.1}{10.233.72.1:9300}{dim} with JoinRequest{sourceNode={cldash-es-node1}{THYoUU8dSEycjJ_09nvebw}{7Ui6jtwVS7OPTOJXUsergQ}{10.233.72.1}{10.233.72.1:9300}{dim}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={cldash-es-node1}{THYoUU8dSEycjJ_09nvebw}{7Ui6jtwVS7OPTOJXUsergQ}{10.233.72.1}{10.233.72.1:9300}{dim}, targetNode={cldash-es-node1}{THYoUU8dSEycjJ_09nvebw}{7Ui6jtwVS7OPTOJXUsergQ}{10.233.72.1}{10.233.72.1:9300}{dim}}]} org.elasticsearch.transport.RemoteTransportException: [cldash-es-node1][10.233.72.1:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 1 does not match current term 2 at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.2.jar:7.3.2] at java.util.Optional.ifPresent(Optional.java:183) ~[?:?] at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.2.jar:7.3.2] at com.amazon.opendistro.elasticsearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?] at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceivedDecorate(OpenDistroSecuritySSLRequestHandler.java:164) ~[?:?] at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityRequestHandler.messageReceivedDecorate(OpenDistroSecurityRequestHandler.java:171) ~[?:?] at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceived(OpenDistroSecuritySSLRequestHandler.java:86) ~[?:?] at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin$7$1.messageReceived(OpenDistroSecurityPlugin.java:635) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2021-04-14T02:13:07,358][INFO ][o.e.c.s.MasterService ] [cldash-es-node1] elected-as-master ([1] nodes joined)[{cldash-es-node1}{THYoUU8dSEycjJ_09nvebw}{7Ui6jtwVS7OPTOJXUsergQ}{10.233.72.1}{10.233.72.1:9300}{dim} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 1, reason: master node changed {previous [], current [{cldash-es-node1}{THYoUU8dSEycjJ_09nvebw}{7Ui6jtwVS7OPTOJXUsergQ}{10.233.72.1}{10.233.72.1:9300}{dim}]} [2021-04-14T02:13:07,546][INFO ][o.e.c.c.CoordinationState] [cldash-es-node1] cluster UUID set to [3sIPESxlSy--mndngzj34w] [2021-04-14T02:13:07,583][INFO ][o.e.c.s.ClusterApplierService] [cldash-es-node1] master node changed {previous [], current [{cldash-es-node1}{THYoUU8dSEycjJ_09nvebw}{7Ui6jtwVS7OPTOJXUsergQ}{10.233.72.1}{10.233.72.1:9300}{dim}]}, term: 2, version: 1, reason: Publication{term=2, version=1} [2021-04-14T02:13:08,066][INFO ][o.e.h.AbstractHttpServerTransport] [cldash-es-node1] publish_address {10.233.72.1:9200}, bound_addresses {0.0.0.0:9200} [2021-04-14T02:13:08,067][INFO ][o.e.n.Node ] [cldash-es-node1] started [2021-04-14T02:13:08,067][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] Node started [2021-04-14T02:13:08,067][INFO ][c.a.o.s.c.ConfigurationRepository] [cldash-es-node1] Check if .opendistro_security index exists ... [2021-04-14T02:13:08,068][INFO ][c.a.o.s.c.ConfigurationRepository] [cldash-es-node1] .opendistro_security index does not exist yet, so we create a default config [2021-04-14T02:13:08,068][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [cldash-es-node1] 4 Open Distro Security modules loaded so far: [Module [type=DLSFLS, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.OpenDistroSecurityFlsDlsIndexSearcherWrapper], Module [type=AUDITLOG, implementing class=com.amazon.opendistroforelasticsearch.security.auditlog.impl.AuditLogImpl], Module [type=MULTITENANCY, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.amazon.opendistroforelasticsearch.security.dlic.rest.api.OpenDistroSecurityRestApiActions]] [2021-04-14T02:13:08,068][INFO ][c.a.o.s.c.ConfigurationRepository] [cldash-es-node1] Background init thread started. Install default config?: true WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.MasterServiceEventMetrics (file:/usr/share/elasticsearch/plugins/opendistro_performance_analyzer/opendistro_performance_analyzer-1.3.0.0.jar) to field java.util.concurrent.ThreadPoolExecutor.workers WARNING: Please consider reporting this to the maintainers of com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.MasterServiceEventMetrics WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [2021-04-14T02:13:08,158][INFO ][c.a.o.s.c.ConfigurationRepository] [cldash-es-node1] Will create .opendistro_security index so we can apply default config [2021-04-14T02:13:08,249][INFO ][o.e.g.GatewayService ] [cldash-es-node1] recovered [0] indices into cluster_state [2021-04-14T02:13:08,555][INFO ][o.e.c.m.MetaDataCreateIndexService] [cldash-es-node1] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1], mappings [] [2021-04-14T02:13:08,567][INFO ][o.e.c.r.a.AllocationService] [cldash-es-node1] updating number_of_replicas to [0] for indices [.opendistro_security] [2021-04-14T02:13:09,674][INFO ][o.e.c.r.a.AllocationService] [cldash-es-node1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]] ...]). [2021-04-14T02:13:09,847][INFO ][c.a.o.s.c.ConfigurationRepository] [cldash-es-node1] Index .opendistro_security created?: true [2021-04-14T02:13:09,850][INFO ][c.a.o.s.s.ConfigHelper ] [cldash-es-node1] Will update 'CONFIG' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml [2021-04-14T02:13:10,183][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.opendistro_security/MXelbfjhSGa2NbbX1Bd11w] create_mapping [_doc] [2021-04-14T02:13:10,720][INFO ][c.a.o.s.s.ConfigHelper ] [cldash-es-node1] Will update 'ROLES' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml [2021-04-14T02:13:10,759][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.opendistro_security/MXelbfjhSGa2NbbX1Bd11w] update_mapping [_doc] [2021-04-14T02:13:11,051][INFO ][c.a.o.s.s.ConfigHelper ] [cldash-es-node1] Will update 'ROLESMAPPING' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles_mapping.yml [2021-04-14T02:13:11,074][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.opendistro_security/MXelbfjhSGa2NbbX1Bd11w] update_mapping [_doc] [2021-04-14T02:13:11,444][INFO ][c.a.o.s.s.ConfigHelper ] [cldash-es-node1] Will update 'INTERNALUSERS' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml [2021-04-14T02:13:11,466][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.opendistro_security/MXelbfjhSGa2NbbX1Bd11w] update_mapping [_doc] [2021-04-14T02:13:12,158][INFO ][c.a.o.s.s.ConfigHelper ] [cldash-es-node1] Will update 'ACTIONGROUPS' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/action_groups.yml [2021-04-14T02:13:12,177][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.opendistro_security/MXelbfjhSGa2NbbX1Bd11w] update_mapping [_doc] [2021-04-14T02:13:12,436][INFO ][c.a.o.s.s.ConfigHelper ] [cldash-es-node1] Will update 'TENANTS' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/tenants.yml [2021-04-14T02:13:12,453][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.opendistro_security/MXelbfjhSGa2NbbX1Bd11w] update_mapping [_doc] [2021-04-14T02:13:12,715][INFO ][c.a.o.s.c.ConfigurationRepository] [cldash-es-node1] Default config applied [2021-04-14T02:13:13,058][INFO ][c.a.o.s.c.ConfigurationRepository] [cldash-es-node1] Node 'cldash-es-node1' initialized [2021-04-14T02:18:06,076][INFO ][c.a.o.j.s.JobSweeper ] [cldash-es-node1] Running full sweep [2021-04-14T02:19:46,589][INFO ][o.e.c.m.MetaDataCreateIndexService] [cldash-es-node1] [security-auditlog-2021.04.14] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings [] [2021-04-14T02:19:46,960][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [security-auditlog-2021.04.14/J5WTEbRqQdG2StJZ4PwF5g] create_mapping [_doc] [2021-04-14T02:23:06,078][INFO ][c.a.o.j.s.JobSweeper ] [cldash-es-node1] Running full sweep [2021-04-14T02:23:44,093][INFO ][o.e.c.m.MetaDataIndexTemplateService] [cldash-es-node1] adding template [logstash] for index patterns [logstash-*] [2021-04-14T02:23:44,244][INFO ][o.e.c.m.MetaDataIndexTemplateService] [cldash-es-node1] adding template [logstash] for index patterns [logstash-*] [2021-04-14T02:23:44,277][INFO ][o.e.c.m.MetaDataIndexTemplateService] [cldash-es-node1] adding template [logstash] for index patterns [logstash-*] [2021-04-14T02:23:55,353][INFO ][o.e.c.m.MetaDataIndexTemplateService] [cldash-es-node1] adding template [tenant_template] for index patterns [.kibana_-*_*, .kibana_0*_*, .kibana_1*_*, .kibana_2*_*, .kibana_3*_*, .kibana_4*_*, .kibana_5*_*, .kibana_6*_*, .kibana_7*_*, .kibana_8*_*, .kibana_9*_*] [2021-04-14T02:23:55,549][INFO ][o.e.c.m.MetaDataCreateIndexService] [cldash-es-node1] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc] [2021-04-14T02:23:55,550][INFO ][o.e.c.r.a.AllocationService] [cldash-es-node1] updating number_of_replicas to [0] for indices [.kibana_1] [2021-04-14T02:23:57,069][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [security-auditlog-2021.04.14/J5WTEbRqQdG2StJZ4PwF5g] update_mapping [_doc] [2021-04-14T02:24:02,770][INFO ][o.e.c.m.MetaDataCreateIndexService] [cldash-es-node1] [.kibana_-1404856460_clampadmin] creating index, cause [auto(bulk api)], templates [tenant_template], shards [1]/[1], mappings [_doc] [2021-04-14T02:24:03,146][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.kibana_-1404856460_clampadmin/kmT7WCYUTs6BqGDz-Evq_Q] update_mapping [_doc] [2021-04-14T02:24:06,564][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.kibana_-1404856460_clampadmin/kmT7WCYUTs6BqGDz-Evq_Q] update_mapping [_doc] [2021-04-14T02:24:16,162][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.kibana_-1404856460_clampadmin/kmT7WCYUTs6BqGDz-Evq_Q] update_mapping [_doc] [2021-04-14T02:24:25,658][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.kibana_-1404856460_clampadmin/kmT7WCYUTs6BqGDz-Evq_Q] update_mapping [_doc] [2021-04-14T02:24:37,963][INFO ][o.e.c.m.MetaDataIndexTemplateService] [cldash-es-node1] adding template [tenant_template] for index patterns [.kibana_-*_*, .kibana_0*_*, .kibana_1*_*, .kibana_2*_*, .kibana_3*_*, .kibana_4*_*, .kibana_5*_*, .kibana_6*_*, .kibana_7*_*, .kibana_8*_*, .kibana_9*_*] [2021-04-14T02:24:38,552][INFO ][o.e.c.m.MetaDataCreateIndexService] [cldash-es-node1] [.kibana_-1404856460_clampadmin_2] creating index, cause [api], templates [tenant_template], shards [1]/[1], mappings [_doc] [2021-04-14T02:24:38,553][INFO ][o.e.c.r.a.AllocationService] [cldash-es-node1] updating number_of_replicas to [0] for indices [.kibana_-1404856460_clampadmin_2] [2021-04-14T02:24:38,942][INFO ][o.e.c.m.MetaDataCreateIndexService] [cldash-es-node1] [.kibana_-1404856460_clampadmin_1] creating index, cause [api], templates [tenant_template], shards [1]/[1], mappings [_doc] [2021-04-14T02:24:38,943][INFO ][o.e.c.r.a.AllocationService] [cldash-es-node1] updating number_of_replicas to [0] for indices [.kibana_-1404856460_clampadmin_1] [2021-04-14T02:24:39,994][INFO ][o.e.c.m.MetaDataCreateIndexService] [cldash-es-node1] [.tasks] creating index, cause [auto(task api)], templates [], shards [1]/[1], mappings [task] [2021-04-14T02:24:39,995][INFO ][o.e.c.r.a.AllocationService] [cldash-es-node1] updating number_of_replicas to [0] for indices [.tasks] [2021-04-14T02:24:40,376][INFO ][o.e.t.LoggingTaskListener] [cldash-es-node1] 373 finished with response BulkByScrollResponse[took=603.6ms,timed_out=false,sliceId=null,updated=0,created=20,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]] [2021-04-14T02:24:40,715][INFO ][o.e.c.m.MetaDataDeleteIndexService] [cldash-es-node1] [.kibana_-1404856460_clampadmin/kmT7WCYUTs6BqGDz-Evq_Q] deleting index [2021-04-14T02:24:41,249][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.kibana_-1404856460_clampadmin_2/Mm2gVCDnQnyP0_RfylC70Q] update_mapping [_doc] [2021-04-14T02:24:41,358][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.kibana_-1404856460_clampadmin_2/Mm2gVCDnQnyP0_RfylC70Q] update_mapping [_doc] [2021-04-14T02:24:41,504][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.kibana_-1404856460_clampadmin_2/Mm2gVCDnQnyP0_RfylC70Q] update_mapping [_doc] [2021-04-14T02:24:41,650][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [.kibana_-1404856460_clampadmin_2/Mm2gVCDnQnyP0_RfylC70Q] update_mapping [_doc] [2021-04-14T02:24:51,690][WARN ][o.e.d.a.b.BulkRequestParser] [cldash-es-node1] [types removal] Specifying types in bulk requests is deprecated. [2021-04-14T02:24:51,751][INFO ][o.e.c.m.MetaDataCreateIndexService] [cldash-es-node1] [errors-2021.04.104] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings [] [2021-04-14T02:24:52,125][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [errors-2021.04.104/C2ORp2eOSzWrLX3fqDryYQ] create_mapping [_doc] [2021-04-14T02:24:52,188][INFO ][o.e.c.m.MetaDataMappingService] [cldash-es-node1] [errors-2021.04.104/C2ORp2eOSzWrLX3fqDryYQ] update_mapping [_doc] [2021-04-14T02:28:06,080][INFO ][c.a.o.j.s.JobSweeper ] [cldash-es-node1] Running full sweep [2021-04-14T02:33:06,080][INFO ][c.a.o.j.s.JobSweeper ] [cldash-es-node1] Running full sweep [2021-04-14T02:38:06,081][INFO ][c.a.o.j.s.JobSweeper ] [cldash-es-node1] Running full sweep [2021-04-14T02:43:06,082][INFO ][c.a.o.j.s.JobSweeper ] [cldash-es-node1] Running full sweep [2021-04-14T02:48:06,083][INFO ][c.a.o.j.s.JobSweeper ] [cldash-es-node1] Running full sweep [2021-04-14T02:53:06,084][INFO ][c.a.o.j.s.JobSweeper ] [cldash-es-node1] Running full sweep