> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Hasura

> Hasura GraphQL Engine request, subscription, event trigger, and Postgres connection metrics.

The Hasura integration requires CXDOT Collector 1.13.0 or greater.

[Hasura GraphQL Engine](https://hasura.io/) serves a GraphQL API over your databases
and services. Use the Hasura integration to collect GraphQL request, subscription,
event trigger, and PostgreSQL connection metrics from Hasura GraphQL Engine Enterprise
Edition instances running in your environment.

The Hasura integration supports Hasura GraphQL Engine 2.36 or greater.

## Supported telemetry types

The Hasura integration supports these telemetry types:

| Type | Supported |
| - | - |
| Logs | No |
| Metrics | Yes |
| Traces | No |
| Events | No |

## Prerequisites

The Hasura integration has the following prerequisites:

* Run Hasura GraphQL Engine Enterprise Edition with an active license. Community
  Edition and unlicensed Enterprise Edition don't serve metrics.

* Enable the metrics API on each Hasura instance by adding `metrics` to
  `HASURA_GRAPHQL_ENABLED_APIS`, such as
  `HASURA_GRAPHQL_ENABLED_APIS=metadata,graphql,pgdump,config,metrics`. If the instance sets
  `HASURA_GRAPHQL_METRICS_SECRET`, every request for metrics must present that secret
  as a bearer token. For more information, see
  [Integrate Prometheus with Hasura](https://hasura.io/docs/2.0/observability/enterprise-edition/prometheus/integrate-prometheus-grafana/).
  To confirm that an instance serves metrics, request `/v1/metrics` from a machine
  that can reach it and confirm that the request returns a successful status code.
  For example:

  ```shell theme={null}
  curl -i http://hasura.example.com:8080/v1/metrics
  ```

  If the instance sets `HASURA_GRAPHQL_METRICS_SECRET`, present that value as a
  bearer token:

  ```shell theme={null}
  curl -i -H "Authorization: Bearer METRICS_SECRET" http://hasura.example.com:8080/v1/metrics
  ```

* Make port `8080`, or the port that `HASURA_GRAPHQL_SERVER_PORT` sets, reachable on
  each Hasura instance from the CXDOT Collector.

## Configure

The Hasura integration is enabled by default, but doesn't collect data until you
configure targets. Add a CXDOT integration annotation to your Hasura pods or to a Service
in front of them, or list the Hasura instances to monitor.

To configure the Hasura integration, follow these steps:

1. To collect from Hasura pods in Kubernetes, add a CXDOT integration annotation to the
   pod template. The annotation key must end with the container name. For example, add
   the following annotation to your Hasura workload:

   ```yaml theme={null}
   spec:
     template:
       metadata:
         annotations:
           integrations.cxdot.paloaltonetworks.com/hasura: |
             schema_version: v1.16
             hasura:
               endpoints:
                 - endpoint: ${target.host}:${target.port.http}
       spec:
         containers:
           - name: hasura
             image: hasura/graphql-engine
             ports:
               - name: http
                 containerPort: 8080
   ```

   The Collector replaces `${target.host}` with the pod IP address and
   `${target.port.http}` with the container port named `http`. Metrics from each
   annotated pod include that pod's Kubernetes metadata. This annotation sets no
   metrics secret, so use it for instances that don't set
   `HASURA_GRAPHQL_METRICS_SECRET`, or see the step about the metrics secret for
   annotated pods.

2. Alternatively, to collect from Hasura instances outside your Kubernetes cluster or
   at addresses that you choose, list each instance in `endpoints`. For example, add
   the following to the Helm values file for your CXDOT Collector:

   ```yaml theme={null}
   collection:
     cxdot:
       integrations:
         hasura:
           endpoints:
             - endpoint: hasura-1.example.com:8080
             - endpoint: hasura-2.example.com:8080
             - endpoint: https://hasura-3.example.com/v1/metrics
   ```

   When you list endpoints, the integration collects from exactly those endpoints
   instead of from annotated pods and Services. In Kubernetes, the elected
   cluster Collector collects from each endpoint that you list, one time rather than
   once per node. For more information, see
   [architecture](https://docs-xcor.paloaltonetworks.com/ingest/xcor/collector/install/kubernetes/architecture.md).

   List an address that reaches one Hasura instance. Each request to an address that
   balances across replicas, such as a Kubernetes Service in front of several pods,
   reaches a different replica, so the metrics mix the replicas' values. For a
   Hasura deployment with several replicas, list each replica's own address. For
   example, a StatefulSet with a headless Service gives each pod a stable DNS name.
   Metrics from listed endpoints don't carry Kubernetes metadata from the Hasura
   pods. Use the `server.address` and `server.port` attributes to identify each
   instance.

3. Optional: To collect from both annotated pods and listed endpoints, list the
   endpoints in a second integration instance. For example, add the following to the
   Helm values file for your CXDOT Collector:

   ```yaml theme={null}
   collection:
     cxdot:
       integrations:
         hasura: {}
         hasura/edge:
           endpoints:
             - endpoint: hasura-1.example.com:8080
   ```

   The `hasura/edge` key defines a named instance for the listed endpoints. The bare
   `hasura` key lists no endpoints, so the integration continues to collect from
   annotated pods and Services.

4. Optional: If you can't change the pod template, configure the Service in front of
   the instances' pods with a CXDOT integration annotation instead. The
   `integrations.cxdot.paloaltonetworks.com/endpoints` key makes each ready pod behind
   the Service a separate target, so the metrics don't mix the replicas' values. For
   example, add the following annotation to that Service:

   ```yaml theme={null}
   metadata:
     annotations:
       integrations.cxdot.paloaltonetworks.com/endpoints: |
         schema_version: v1.16
         hasura:
           endpoints:
             - endpoint: ${target.host}:${target.port.http}
   spec:
     ports:
       - name: http
         port: 80
         targetPort: 8080
   ```

   The Collector replaces `${target.host}` with the IP address of each pod and
   `${target.port.http}` with the pod port that the Service port named `http` forwards
   to. This annotation sets no metrics secret, so use it for instances that don't set
   `HASURA_GRAPHQL_METRICS_SECRET`. The Collector reads these annotations only when
   `endpoints` is empty.

5. Optional: To supply the metrics secret for listed endpoints, mount the secret from a
   Kubernetes Secret in the namespace of your CXDOT Collector with the chart's
   `clusterCollector.extraVolumes` and
   `clusterCollector.extraVolumeMounts` values, then set `metrics_secret_file` to the
   mounted path. For example:

   ```yaml theme={null}
   clusterCollector:
     extraVolumes:
       - name: hasura-metrics-secret
         secret:
           secretName: hasura-metrics-secret
     extraVolumeMounts:
       - name: hasura-metrics-secret
         mountPath: /etc/cxdot/hasura
         readOnly: true
   collection:
     cxdot:
       integrations:
         hasura:
           metrics_secret_file: /etc/cxdot/hasura/secret
           endpoints:
             - endpoint: hasura-1.example.com:8080
   ```

   Kubernetes mounts each key of the Secret as a file named after the key, so the
   `/etc/cxdot/hasura/secret` path in this example exists only if the Secret
   stores the metrics secret under the key `secret`. For a different key, set
   `metrics_secret_file` to the path for that key. The Collector sends the same
   secret to every endpoint in one integration instance.

6. Optional: Add a named integration instance for listed Hasura deployments that use
   a different metrics secret. Mount each instance's Secret as in the previous step,
   with its own volume and mount path. For example:

   ```yaml theme={null}
   collection:
     cxdot:
       integrations:
         hasura:
           metrics_secret_file: /etc/cxdot/hasura/secret
           endpoints:
             - endpoint: hasura-1.example.com:8080
         hasura/billing:
           metrics_secret_file: /etc/cxdot/hasura-billing/secret
           endpoints:
             - endpoint: hasura-billing.example.com:8080
   ```

   The `hasura/billing` key defines a second integration instance with its own
   endpoints and settings. Its metrics carry the
   `cxdot.integration.instance.name` attribute with the value `billing`.

7. Optional: To collect from annotated pods that set a metrics secret, mount the Secret
   into the node Collector and allow its path. The `metrics_secret_file` setting takes
   the path of a file that the Collector reads, so an annotation can't carry the
   secret itself. The Collector rejects a `hasura` block that names a file outside
   `allowed_collector_file_paths`, which allows no paths by default. The Secret must be
   in the namespace of your CXDOT Collector. For example:

   ```yaml theme={null}
   nodeCollector:
     extraVolumes:
       - name: hasura-metrics-secret
         secret:
           secretName: hasura-metrics-secret
     extraVolumeMounts:
       - name: hasura-metrics-secret
         mountPath: /etc/cxdot/hasura
         readOnly: true
   collection:
     cxdot:
       integrations:
         discovery:
           kubernetes:
             annotation_restrictions:
               allowed_collector_file_paths:
                 - /etc/cxdot/hasura
               custom_pod_host: deny
   ```

   Then set `metrics_secret_file: /etc/cxdot/hasura/secret` in the `hasura` block of
   the pod annotation, where `secret` is the key that holds the metrics secret in the
   Kubernetes Secret. After you allow a path, an annotation on any pod in your cluster
   can name the file, and the Collector sends the secret to that annotation's target.
   The `custom_pod_host: deny` setting limits each pod annotation's target to the pod's
   own IP address, so the secret reaches only annotated pods. Allow only paths that hold
   secrets that you intend annotated pods to use.

8. Optional: If the OpenMetrics integration also scrapes your Hasura pods or Service,
   disable OpenMetrics on that workload to avoid duplicate metrics. For example, set
   `openmetrics: {enabled: false}` in a CXDOT integration annotation on the pod or
   Service. For more information, see
   [autodiscovery](https://docs-xcor.paloaltonetworks.com/ingest/xcor/collector/discover/kubernetes.md).

### Validate

To validate the Hasura integration, follow these steps:

1. In the [Live Telemetry Analyzer](https://docs-xcor.paloaltonetworks.com/investigate/analyze/telemetry-analyzer.md),
   filter for
   `__name__=cxdot.integration.target.health` and
   `cxdot.integration.name=hasura`. Confirm that each Hasura instance reports
   `1` for `cxdot.integration.target.health`, identified by its
   `server.address` and `server.port` attributes.

2. In [Metrics Explorer](https://docs-xcor.paloaltonetworks.com/investigate/querying/metrics/explorer.md),
   run the following query while Hasura serves GraphQL requests:

   ```text theme={null}
   sum by ("server.address", "graphql.operation.type") (rate({"hasura.graphql.requests"}[$__rate_interval]))
   ```

   Confirm that the query returns a time series for each instance and operation type
   that serves requests.

### Troubleshooting

* An endpoint reports `0` for `cxdot.integration.target.health`: Confirm that the
  instance meets the requirements in the Prerequisites section and that
  `metrics_secret_file` holds the instance's current metrics secret. If you set
  `max_samples`, a collection that exceeds the limit also reports `0`.
* Source health metrics don't appear: Hasura reports source health only for
  sources that have a
  [source health check](https://hasura.io/docs/2.0/deployment/health-checks/source-health-check/)
  configured.

## Configuration reference

Configure one Hasura integration instance with the following settings. In Helm values, place
these settings under `collection.cxdot.integrations.hasura`. In a Collector configuration file,
place them under `cxdot.integrations.hasura`.

### Optional settings

* **`enabled`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to enable this Hasura integration instance. If true, the Collector collects metrics
  from the endpoints that this instance is configured for. If false, the Collector doesn't run
  this integration instance.

* **`endpoints`**
  Type: `array of object`. Optional. Default: `[]`.
  Hasura GraphQL Engine endpoints to monitor. Specify a full HTTP or HTTPS URL that includes the
  `/v1/metrics` path, or use `host:port` format to request the `/v1/metrics` path over HTTP. For
  a URL without a path, the Collector requests `/metrics`, which Hasura doesn't serve. Hasura
  exposes metrics only in Enterprise Edition, with `metrics` in `HASURA_GRAPHQL_ENABLED_APIS`.
  Each entry is one target, so list an address that reaches one Hasura instance. An address that
  balances requests across replicas, such as a Kubernetes Service with several pods, reaches a
  different replica on each collection. The integration instance collects only from the
  endpoints listed here.

* **`endpoints[].endpoint`**
  Type: `string`. Required.
  Hasura GraphQL Engine address, as a full HTTP or HTTPS URL or in `host:port` format. For
  `host:port`, the Collector requests the `/v1/metrics` path over HTTP. A full URL must include
  the `/v1/metrics` path, such as `https://hasura.example.com/v1/metrics`. For a URL without a
  path, the Collector requests `/metrics`, which Hasura doesn't serve.

* **`metrics_secret_file`**
  Type: `string`. Optional. Default: empty string.
  Path to a file that holds the value of Hasura's `HASURA_GRAPHQL_METRICS_SECRET`. The Collector
  sends this value as a bearer token on every request. The Collector rereads the file when it
  changes. If Hasura has no metrics secret, omit this setting.

* **`tls`**
  Type: `object`. Optional.
  Transport layer security (TLS) settings for HTTPS endpoints. The Collector verifies
  certificates unless `insecure_skip_verify` is `true`. These settings don't apply to HTTP
  endpoints.

* **`tls.ca_file`**
  Type: `string`. Optional.
  Path to a PEM-encoded certificate authority (CA) certificate file used to verify the Hasura
  server certificate. If set, the Collector trusts only the certificate authorities in this file
  instead of the system certificate authority pool.

* **`tls.insecure_skip_verify`**
  Type: `boolean`. Optional.
  Whether to skip verification of the Hasura server certificate. If true, the connection is
  encrypted but isn't authenticated and is vulnerable to man-in-the-middle attacks. If false,
  the Collector verifies the server certificate.

* **`tls.server_name`**
  Type: `string`. Optional.
  Server name to verify in the Hasura server certificate instead of the endpoint hostname. The
  Collector also sends this name as the TLS server name indication. Use this setting when the
  certificate name differs from the endpoint hostname.

* **`collection_interval`**
  Type: `duration`. Optional. Default: `15s`.
  How often the integration collects metrics from each Hasura endpoint.

* **`timeout`**
  Type: `duration`. Optional. Default: `10s`.
  Maximum time allowed to collect metrics from a Hasura endpoint during one interval. The value
  must be greater than zero.

* **`max_samples`**
  Type: `integer`. Optional. Default: `0`. Minimum: `0`.
  Maximum number of samples accepted from one metric collection. A value of `0`, the default,
  means no limit. A collection that exceeds the limit fails and reports the target as unhealthy
  instead of returning an incomplete set of samples. If your Hasura version labels request
  metrics by operation, the number of samples grows with the number of distinct operations that
  a deployment serves.

* **`metrics`**
  Type: `object`. Optional.
  Settings that control which Hasura metrics the integration collects and how it names them.

* **`metrics.include`**
  Type: `array of string`. Optional. Default: `[]`.
  Regular expressions for metrics to collect in addition to the integration's default metric
  set. Each expression must match a whole metric name as Hasura exposes it, without a `_total`
  suffix. A matched metric outside the default set keeps its exposed name.

* **`metrics.exclude`**
  Type: `array of string`. Optional. Default: `[]`.
  Regular expressions for metrics to drop. Each expression must match a whole metric name as
  Hasura exposes it, without a `_total` suffix, such as `hasura_otel_.*`. Exclusions apply
  before renaming and take precedence over `include`.

* **`metrics.rename`**
  Type: `object`. Optional. Default: `{}`.
  Mapping of metric names as Hasura exposes them, without a `_total` suffix, to the names that
  the Collector reports. These mappings take precedence over the integration's default metric
  names. Renaming a metric also collects it.

* **`labels`**
  Type: `object`. Optional.
  Settings that drop metric samples based on their label values. Rules match the label names
  that Hasura exposes. The integration reports `operation_name` as `graphql.operation.name` and
  `operation_type` as `graphql.operation.type`. It reports the `source_name` label of Postgres
  connection metrics as `db.client.connection.pool.name`. In a rule, use the label name that
  Hasura exposes, not the reported name. Other labels, such as `trigger_name` and the
  `source_name` label of other metrics, keep their names.

* **`labels.sample_drop_rules`**
  Type: `object`. Optional. Default: `{}`.
  Drops a whole sample when one of its labels matches. Each key is a label name. The value is a
  list of regular expressions matched against the label's value. To drop every sample the label
  appears on with a non-empty value, use `".+"`.


## Related topics

- [CXDOT Collector integrations](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.