Supported telemetry types
The Hasura integration supports these telemetry types:Prerequisites
The Hasura integration has the following prerequisites:- Run Hasura GraphQL Engine Enterprise Edition with an active license. Community Edition and unlicensed Enterprise Edition don’t serve metrics.
-
Enable the metrics API on each Hasura instance by adding
metricstoHASURA_GRAPHQL_ENABLED_APIS, such asHASURA_GRAPHQL_ENABLED_APIS=metadata,graphql,pgdump,config,metrics. If the instance setsHASURA_GRAPHQL_METRICS_SECRET, every request for metrics must present that secret as a bearer token. For more information, see Integrate Prometheus with Hasura. To confirm that an instance serves metrics, request/v1/metricsfrom a machine that can reach it and confirm that the request returns a successful status code. For example:If the instance setsHASURA_GRAPHQL_METRICS_SECRET, present that value as a bearer token: -
Make port
8080, or the port thatHASURA_GRAPHQL_SERVER_PORTsets, reachable on each Hasura instance from the CXDOT Collector.
Configure
The Hasura integration is enabled by default, but doesn’t collect data until you configure targets. Add a CXDOT integration annotation to your Hasura pods or to a Service in front of them, or list the Hasura instances to monitor. To configure the Hasura integration, follow these steps:-
To collect from Hasura pods in Kubernetes, add a CXDOT integration annotation to the
pod template. The annotation key must end with the container name. For example, add
the following annotation to your Hasura workload:
The Collector replaces
${target.host}with the pod IP address and${target.port.http}with the container port namedhttp. Metrics from each annotated pod include that pod’s Kubernetes metadata. This annotation sets no metrics secret, so use it for instances that don’t setHASURA_GRAPHQL_METRICS_SECRET, or see the step about the metrics secret for annotated pods. -
Alternatively, to collect from Hasura instances outside your Kubernetes cluster or
at addresses that you choose, list each instance in
endpoints. For example, add the following to the Helm values file for your CXDOT Collector:When you list endpoints, the integration collects from exactly those endpoints instead of from annotated pods and Services. In Kubernetes, the elected cluster Collector collects from each endpoint that you list, one time rather than once per node. For more information, see architecture. List an address that reaches one Hasura instance. Each request to an address that balances across replicas, such as a Kubernetes Service in front of several pods, reaches a different replica, so the metrics mix the replicas’ values. For a Hasura deployment with several replicas, list each replica’s own address. For example, a StatefulSet with a headless Service gives each pod a stable DNS name. Metrics from listed endpoints don’t carry Kubernetes metadata from the Hasura pods. Use theserver.addressandserver.portattributes to identify each instance. -
Optional: To collect from both annotated pods and listed endpoints, list the
endpoints in a second integration instance. For example, add the following to the
Helm values file for your CXDOT Collector:
The
hasura/edgekey defines a named instance for the listed endpoints. The barehasurakey lists no endpoints, so the integration continues to collect from annotated pods and Services. -
Optional: If you can’t change the pod template, configure the Service in front of
the instances’ pods with a CXDOT integration annotation instead. The
integrations.cxdot.paloaltonetworks.com/endpointskey makes each ready pod behind the Service a separate target, so the metrics don’t mix the replicas’ values. For example, add the following annotation to that Service:The Collector replaces${target.host}with the IP address of each pod and${target.port.http}with the pod port that the Service port namedhttpforwards to. This annotation sets no metrics secret, so use it for instances that don’t setHASURA_GRAPHQL_METRICS_SECRET. The Collector reads these annotations only whenendpointsis empty. -
Optional: To supply the metrics secret for listed endpoints, mount the secret from a
Kubernetes Secret in the namespace of your CXDOT Collector with the chart’s
clusterCollector.extraVolumesandclusterCollector.extraVolumeMountsvalues, then setmetrics_secret_fileto the mounted path. For example:Kubernetes mounts each key of the Secret as a file named after the key, so the/etc/cxdot/hasura/secretpath in this example exists only if the Secret stores the metrics secret under the keysecret. For a different key, setmetrics_secret_fileto the path for that key. The Collector sends the same secret to every endpoint in one integration instance. -
Optional: Add a named integration instance for listed Hasura deployments that use
a different metrics secret. Mount each instance’s Secret as in the previous step,
with its own volume and mount path. For example:
The
hasura/billingkey defines a second integration instance with its own endpoints and settings. Its metrics carry thecxdot.integration.instance.nameattribute with the valuebilling. -
Optional: To collect from annotated pods that set a metrics secret, mount the Secret
into the node Collector and allow its path. The
metrics_secret_filesetting takes the path of a file that the Collector reads, so an annotation can’t carry the secret itself. The Collector rejects ahasurablock that names a file outsideallowed_collector_file_paths, which allows no paths by default. The Secret must be in the namespace of your CXDOT Collector. For example:Then setmetrics_secret_file: /etc/cxdot/hasura/secretin thehasurablock of the pod annotation, wheresecretis the key that holds the metrics secret in the Kubernetes Secret. After you allow a path, an annotation on any pod in your cluster can name the file, and the Collector sends the secret to that annotation’s target. Thecustom_pod_host: denysetting limits each pod annotation’s target to the pod’s own IP address, so the secret reaches only annotated pods. Allow only paths that hold secrets that you intend annotated pods to use. -
Optional: If the OpenMetrics integration also scrapes your Hasura pods or Service,
disable OpenMetrics on that workload to avoid duplicate metrics. For example, set
openmetrics: {enabled: false}in a CXDOT integration annotation on the pod or Service. For more information, see autodiscovery.
Validate
To validate the Hasura integration, follow these steps:-
In the Live Telemetry Analyzer,
filter for
__name__=cxdot.integration.target.healthandcxdot.integration.name=hasura. Confirm that each Hasura instance reports1forcxdot.integration.target.health, identified by itsserver.addressandserver.portattributes. -
In Metrics Explorer,
run the following query while Hasura serves GraphQL requests:
Confirm that the query returns a time series for each instance and operation type that serves requests.
Troubleshooting
- An endpoint reports
0forcxdot.integration.target.health: Confirm that the instance meets the requirements in the Prerequisites section and thatmetrics_secret_fileholds the instance’s current metrics secret. If you setmax_samples, a collection that exceeds the limit also reports0. - Source health metrics don’t appear: Hasura reports source health only for sources that have a source health check configured.
Configuration reference
Configure one Hasura integration instance with the following settings. In Helm values, place these settings undercollection.cxdot.integrations.hasura. In a Collector configuration file,
place them under cxdot.integrations.hasura.
Optional settings
-
enabledType:boolean. Optional. Default:true. Whether to enable this Hasura integration instance. If true, the Collector collects metrics from the endpoints that this instance is configured for. If false, the Collector doesn’t run this integration instance. -
endpointsType:array of object. Optional. Default:[]. Hasura GraphQL Engine endpoints to monitor. Specify a full HTTP or HTTPS URL that includes the/v1/metricspath, or usehost:portformat to request the/v1/metricspath over HTTP. For a URL without a path, the Collector requests/metrics, which Hasura doesn’t serve. Hasura exposes metrics only in Enterprise Edition, withmetricsinHASURA_GRAPHQL_ENABLED_APIS. Each entry is one target, so list an address that reaches one Hasura instance. An address that balances requests across replicas, such as a Kubernetes Service with several pods, reaches a different replica on each collection. The integration instance collects only from the endpoints listed here. -
endpoints[].endpointType:string. Required. Hasura GraphQL Engine address, as a full HTTP or HTTPS URL or inhost:portformat. Forhost:port, the Collector requests the/v1/metricspath over HTTP. A full URL must include the/v1/metricspath, such ashttps://hasura.example.com/v1/metrics. For a URL without a path, the Collector requests/metrics, which Hasura doesn’t serve. -
metrics_secret_fileType:string. Optional. Default: empty string. Path to a file that holds the value of Hasura’sHASURA_GRAPHQL_METRICS_SECRET. The Collector sends this value as a bearer token on every request. The Collector rereads the file when it changes. If Hasura has no metrics secret, omit this setting. -
tlsType:object. Optional. Transport layer security (TLS) settings for HTTPS endpoints. The Collector verifies certificates unlessinsecure_skip_verifyistrue. These settings don’t apply to HTTP endpoints. -
tls.ca_fileType:string. Optional. Path to a PEM-encoded certificate authority (CA) certificate file used to verify the Hasura server certificate. If set, the Collector trusts only the certificate authorities in this file instead of the system certificate authority pool. -
tls.insecure_skip_verifyType:boolean. Optional. Whether to skip verification of the Hasura server certificate. If true, the connection is encrypted but isn’t authenticated and is vulnerable to man-in-the-middle attacks. If false, the Collector verifies the server certificate. -
tls.server_nameType:string. Optional. Server name to verify in the Hasura server certificate instead of the endpoint hostname. The Collector also sends this name as the TLS server name indication. Use this setting when the certificate name differs from the endpoint hostname. -
collection_intervalType:duration. Optional. Default:15s. How often the integration collects metrics from each Hasura endpoint. -
timeoutType:duration. Optional. Default:10s. Maximum time allowed to collect metrics from a Hasura endpoint during one interval. The value must be greater than zero. -
max_samplesType:integer. Optional. Default:0. Minimum:0. Maximum number of samples accepted from one metric collection. A value of0, the default, means no limit. A collection that exceeds the limit fails and reports the target as unhealthy instead of returning an incomplete set of samples. If your Hasura version labels request metrics by operation, the number of samples grows with the number of distinct operations that a deployment serves. -
metricsType:object. Optional. Settings that control which Hasura metrics the integration collects and how it names them. -
metrics.includeType:array of string. Optional. Default:[]. Regular expressions for metrics to collect in addition to the integration’s default metric set. Each expression must match a whole metric name as Hasura exposes it, without a_totalsuffix. A matched metric outside the default set keeps its exposed name. -
metrics.excludeType:array of string. Optional. Default:[]. Regular expressions for metrics to drop. Each expression must match a whole metric name as Hasura exposes it, without a_totalsuffix, such ashasura_otel_.*. Exclusions apply before renaming and take precedence overinclude. -
metrics.renameType:object. Optional. Default:{}. Mapping of metric names as Hasura exposes them, without a_totalsuffix, to the names that the Collector reports. These mappings take precedence over the integration’s default metric names. Renaming a metric also collects it. -
labelsType:object. Optional. Settings that drop metric samples based on their label values. Rules match the label names that Hasura exposes. The integration reportsoperation_nameasgraphql.operation.nameandoperation_typeasgraphql.operation.type. It reports thesource_namelabel of Postgres connection metrics asdb.client.connection.pool.name. In a rule, use the label name that Hasura exposes, not the reported name. Other labels, such astrigger_nameand thesource_namelabel of other metrics, keep their names. -
labels.sample_drop_rulesType:object. Optional. Default:{}. Drops a whole sample when one of its labels matches. Each key is a label name. The value is a list of regular expressions matched against the label’s value. To drop every sample the label appears on with a non-empty value, use".+".