> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Elasticsearch

> Collect cluster, node, index, operating system, process, and JVM metrics from Elasticsearch.

The Elasticsearch integration requires CXDOT Collector 1.7.0 or greater.

[Elasticsearch](https://www.elastic.co/elasticsearch) is a distributed search and
analytics engine. Use the Elasticsearch integration with the CXDOT Collector to collect
cluster, node, operating system, process, and Java Virtual Machine metrics from
Elasticsearch servers running in your environment.

The Elasticsearch integration supports Elasticsearch 7.9 or later.

## Supported telemetry types

The Elasticsearch integration supports these telemetry types:

| Type | Supported |
| - | - |
| Logs | No |
| Metrics | Yes |
| Traces | No |
| Events | No |

## Prerequisites

The Elasticsearch integration has the following prerequisites:

* If Elasticsearch security is enabled, create a monitoring user with the `monitor`
  or `manage` cluster privilege. The `monitor` privilege provides the required
  read-only access. For more information, see
  [Elasticsearch security privileges](https://www.elastic.co/docs/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges).
* Make the Elasticsearch HTTP API reachable from the CXDOT Collector.

## Configure

The Elasticsearch integration is enabled by default. You can configure targets either by
annotating pods in your Kubernetes cluster or by listing static endpoints in your Helm
values. To configure the integration, follow these steps:

1. To collect from Elasticsearch pods in Kubernetes, add a CXDOT integration annotation
   to the pod template. The annotation key must end with the container name, and its
   value can reference credentials and a certificate authority stored in a Kubernetes
   Secret in the same namespace. For example, add the following annotation to your
   Elasticsearch workload:

   ```yaml theme={null}
   spec:
     template:
       metadata:
         annotations:
           integrations.cxdot.paloaltonetworks.com/elasticsearch: |
             schema_version: v1.14
             elasticsearch:
               endpoints:
                 - endpoint: https://${target.host}:${target.port.http}
               username: cxdot
               password: ${k8s_secret:elasticsearch-monitoring/password}
               tls:
                 ca_pem: ${k8s_secret:elasticsearch-monitoring/ca.crt}
       spec:
         containers:
           - name: elasticsearch
             image: docker.elastic.co/elasticsearch/elasticsearch:9.1.4
             ports:
               - name: http
                 containerPort: 9200
   ```

   The Collector replaces `${target.host}` with the pod IP address,
   `${target.port.http}` with the container port named `http`, and
   `${k8s_secret:elasticsearch-monitoring/password}` and
   `${k8s_secret:elasticsearch-monitoring/ca.crt}` with the `password` and `ca.crt` keys
   of the `elasticsearch-monitoring` Secret. Elasticsearch enables security and HTTPS by
   default, so the endpoint uses `https://` and `ca.crt` holds the certificate authority
   that signed the Elasticsearch HTTP certificate. To allow the Collector to read the
   Secret, add the pod's namespace to `apiServer.metadataServer.secretAllowlist` in the
   Helm values for your CXDOT Collector.

   The Collector also discovers Elasticsearch pods that carry the
   `app.kubernetes.io/name: elasticsearch` label and expose an unsecured HTTP API on
   port `9200`. A CXDOT integration annotation takes precedence over the label for its
   container. To stop collecting from a container, set `elasticsearch: {enabled: false}`
   in the annotation. Metrics from each discovered pod include that pod's Kubernetes
   metadata. For more information, see
   [autodiscovery](https://docs-xcor.paloaltonetworks.com/ingest/xcor/collector/discover/kubernetes.md).

2. Optional: To collect from Elasticsearch servers outside your Kubernetes cluster,
   configure static targets. Supply the password
   through your deployment's secret management. For example, add the following to
   the `values.yaml` for your CXDOT Collector Helm chart:

   ```yaml theme={null}
   collection:
     cxdot:
       integrations:
         elasticsearch:
           endpoints:
             - endpoint: https://elasticsearch.example.com:9200
           username: cxdot
           password: "${env:ELASTICSEARCH_PASSWORD}"
   ```

   When you configure static targets, this integration instance collects from
   exactly those targets instead of discovered targets. In Kubernetes cluster mode, the
   elected cluster Collector scrapes static targets. For more information, see
   [architecture](https://docs-xcor.paloaltonetworks.com/ingest/xcor/collector/install/kubernetes/architecture.md).

3. Optional: Configure a second integration instance to collect from both annotated
   pods and static targets. For example, add the following to the `values.yaml` for
   your CXDOT Collector Helm chart:

   ```yaml theme={null}
   collection:
     cxdot:
       integrations:
         elasticsearch: {}
         elasticsearch/edge:
           endpoints:
             - endpoint: elasticsearch.example.com:9200
   ```

   The `elasticsearch/edge` key defines a named instance for the static target. The
   bare `elasticsearch` key continues to discover pods.

4. Optional: Disable the integration. For example, add the following to the
   `values.yaml` for your CXDOT Collector Helm chart:

   ```yaml theme={null}
   collection:
     cxdot:
       integrations:
         elasticsearch:
           enabled: false
   ```

### Validate

To validate the Elasticsearch integration, follow these steps:

1. In
   [Live Telemetry Analyzer](https://docs-xcor.paloaltonetworks.com/investigate/analyze/telemetry-analyzer.md),
   filter for `__name__=cxdot.integration.target.health` and
   `cxdot.integration.name=elasticsearch`. Confirm that each reachable Elasticsearch
   target reports `1`. A value of `0` means the Collector hit an error collecting from
   that target. The `cxdot.integration.target`, `server.address`, and `server.port`
   attributes identify the target.

2. In
   [Metrics Explorer](https://docs-xcor.paloaltonetworks.com/investigate/querying/metrics/explorer.md),
   run the following query after Elasticsearch has indexed documents:

   ```text theme={null}
   sum by ("server.address", "server.port") ({"elasticsearch.node.documents", "state"="active"})
   ```

   With the default `nodes: [_local]` setting, confirm that the query returns one time
   series per target and that each value is greater than `0` once the cluster has
   indexed documents. Each value counts active documents on the Elasticsearch node that
   answered that target's scrape, not the whole cluster.

## Configuration reference

Configure one Elasticsearch integration instance with the following settings. In Helm values,
place these settings under `collection.cxdot.integrations.elasticsearch`. In a Collector
configuration file, place them under `cxdot.integrations.elasticsearch`.

### Optional settings

* **`enabled`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to enable this Elasticsearch integration instance. If true, the Collector collects
  Elasticsearch metrics. If false, the Collector doesn't run this integration instance.

* **`endpoints`**
  Type: `array of object`. Optional. Default: `[]`.
  Elasticsearch targets. Each `endpoint` is either `host:port` for an HTTP connection or a full
  HTTP or HTTPS URL. When this list contains targets, this integration instance collects from
  exactly those targets and doesn't use discovery. When the list is empty or omitted, the
  integration discovers Elasticsearch targets from Kubernetes pod labels and discovery
  annotations. Configure a second named integration instance to use both listed targets and
  discovery.

* **`endpoints[].endpoint`**
  Type: `string`. Required.
  Elasticsearch target as either `host:port` for an HTTP connection or a full HTTP or HTTPS URL.

* **`username`**
  Type: `string`. Optional.
  Username for HTTP basic authentication to the targets in `endpoints`. Set `username` and
  `password` together. Targets discovered through discovery annotations use the credentials in
  their annotations.

* **`password`**
  Type: `string`. Optional.
  Password for HTTP basic authentication to the targets in `endpoints`. Set `username` and
  `password` together. The Collector masks this value in diagnostic output, logs, and errors.
  Targets discovered through discovery annotations use the credentials in their annotations.

* **`collection_interval`**
  Type: `duration`. Optional. Default: `15s`.
  How often the Collector collects metrics from each Elasticsearch target.

* **`timeout`**
  Type: `duration`. Optional. Default: `10s`.
  Maximum time allowed to collect metrics from one Elasticsearch target. The value must not
  exceed `collection_interval`.

* **`nodes`**
  Type: `array of string`. Optional. Default: `["_local"]`.
  Elasticsearch node selectors to collect. Use `_local` to collect the node that receives the
  request, `_all` to collect every node in the cluster, or Elasticsearch node names and selector
  expressions. An empty list disables node metrics.

* **`skip_cluster_metrics`**
  Type: `boolean`. Optional. Default: `false`.
  If true, don't collect cluster metrics, including pending task and index template metrics. If
  false, collect cluster metrics.

* **`indices`**
  Type: `array of string`. Optional. Default: `[]`.
  Elasticsearch index names and index selector expressions to collect. Use `_all` to collect
  every index. An empty list disables index metrics.

* **`pending_tasks`**
  Type: `boolean`. Optional. Default: `true`.
  If true, collect the number, priority, and average queue time of pending cluster tasks. If
  false, don't request pending task details or emit pending task metrics.

* **`tls`**
  Type: `object`. Optional.
  Transport Layer Security (TLS) settings for the HTTPS targets in `endpoints`. The Collector
  verifies server certificates by default. Targets discovered through annotations use the TLS
  settings in their annotations.

* **`tls.ca_file`**
  Type: `string`. Optional.
  Path to a PEM-encoded certificate authority (CA) certificate file used to verify the
  Elasticsearch server certificate. If omitted, the Collector uses the system certificate
  authority pool.

* **`tls.ca_pem`**
  Type: `string`. Optional.
  PEM-encoded certificate authority certificate content used to verify the Elasticsearch server
  certificate.

* **`tls.cert_file`**
  Type: `string`. Optional.
  Path to the PEM-encoded client certificate file to present when the Elasticsearch endpoint
  requires mutual TLS authentication.

* **`tls.cert_pem`**
  Type: `string`. Optional.
  PEM-encoded client certificate content to present when the Elasticsearch endpoint requires
  mutual TLS authentication.

* **`tls.cipher_suites`**
  Type: `array of string`. Optional.
  TLS cipher suites that the Collector can use, in preference order. If omitted, the Collector
  uses the secure defaults from the Go TLS library. For supported names, see
  [https://go.dev/src/crypto/tls/cipher\_suites.go](https://go.dev/src/crypto/tls/cipher_suites.go).

* **`tls.curve_preferences`**
  Type: `array of string`. Optional.
  Elliptic curves that the Collector can use for an elliptic curve Diffie-Hellman ephemeral
  handshake, in preference order. If omitted, the Collector uses the Go TLS library defaults.

* **`tls.include_system_ca_certs_pool`**
  Type: `boolean`. Optional.
  Whether to add certificates from the system certificate authority pool to the configured
  certificate authorities. If true, the Collector trusts both sources. If false, the Collector
  trusts only the configured certificate authorities.

* **`tls.insecure`**
  Type: `boolean`. Optional.
  Whether to disable transport security. If true, the Collector uses an unencrypted connection.
  If false, the Collector uses TLS for HTTPS endpoints.

* **`tls.insecure_skip_verify`**
  Type: `boolean`. Optional.
  Whether to skip verification of the Elasticsearch server certificate. If true, the connection
  is encrypted but isn't authenticated and is vulnerable to man-in-the-middle attacks. If false,
  the Collector verifies the certificate.

* **`tls.key_file`**
  Type: `string`. Optional.
  Path to the PEM-encoded client private key file used with `cert_file` for mutual TLS
  authentication.

* **`tls.key_pem`**
  Type: `string`. Optional.
  PEM-encoded client private key content used with `cert_pem` for mutual TLS authentication.

* **`tls.max_version`**
  Type: `string`. Optional.
  Maximum TLS protocol version the Collector can use. If omitted, the Collector uses the maximum
  version supported by the Go TLS library.

* **`tls.min_version`**
  Type: `string`. Optional.
  Minimum TLS protocol version the Collector can use. If omitted, the minimum version is TLS
  1.2.

* **`tls.reload_interval`**
  Type: `duration`. Optional.
  How often the Collector reloads certificate and key files. If omitted, the Collector doesn't
  reload the files.

* **`tls.server_name_override`**
  Type: `string`. Optional.
  Server name to verify in the Elasticsearch certificate instead of the endpoint hostname. Use
  this setting when the certificate name differs from the endpoint hostname.

* **`tls.tpm`**
  Type: `object`. Optional.
  Trusted platform module settings for the TLS private key.

* **`tls.tpm.auth`**
  Type: `string`. Optional.
  Authorization value for the trusted platform module key.

* **`tls.tpm.enabled`**
  Type: `boolean`. Optional.
  Whether to use a trusted platform module for the TLS private key. If true, the Collector loads
  the key from the configured device or socket. If false, the Collector uses the configured key
  file or in-memory key.

* **`tls.tpm.owner_auth`**
  Type: `string`. Optional.
  Owner authorization value for the trusted platform module.

* **`tls.tpm.path`**
  Type: `string`. Optional.
  Path to the trusted platform module device or Unix domain socket, such as `/dev/tpm0` or
  `/dev/tpmrm0`.


## Related topics

- [Elasticsearch destination plugin](https://docs-xcor.paloaltonetworks.com/ingest/pipeline/plugins/destination-plugins/elasticsearch.md)
- [Elasticsearch HTTP source plugin](https://docs-xcor.paloaltonetworks.com/ingest/pipeline/plugins/source-plugins/elasticsearch-http.md)
- [Route logs through Palo Alto Networks Telemetry Pipeline](https://docs-xcor.paloaltonetworks.com/ingest/logs/pipeline-logs.md)
- [Send telemetry data with destination plugins](https://docs-xcor.paloaltonetworks.com/ingest/pipeline/plugins/destination-plugins.md)
- [Ingest telemetry data with source plugins](https://docs-xcor.paloaltonetworks.com/ingest/pipeline/plugins/source-plugins.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.