Supported telemetry types
The Kafka Consumer integration supports these telemetry types:Prerequisites
The Kafka Consumer integration has the following prerequisites:-
Make every broker in the cluster reachable from the CXDOT Collector at the address the
broker advertises. The Collector connects to the cluster through one bootstrap address,
learns the other brokers from the cluster metadata, and then connects to each broker at
its advertised listener. If the Collector can’t reach one of the brokers, it reports no
consumer group metrics for the cluster in that collection interval.
To list the advertised addresses, run the
kafka-broker-api-versions.shtool from thebindirectory of a Kafka installation, on a host or pod with the same network access as the CXDOT Collector:ReplaceBOOTSTRAP_ADDRESSwith thehost:portthat the integration connects to. The output lists each broker by its advertised address and ID. Confirm that no broker’s entry reportsERROR. For a cluster that requires authentication or TLS, pass the client settings in a properties file with the--command-configoption. -
If your cluster requires authentication, prepare credentials for SASL
PLAIN,SCRAM-SHA-256, orSCRAM-SHA-512, for Amazon Managed Streaming for Apache Kafka (MSK) Identity and Access Management (IAM) authentication, or for Kerberos. If your cluster enforces authorization, grant the principal that the Collector authenticates as theDescribeoperation on the topics and consumer groups to report. For more information, see Authorization in the Kafka documentation. With MSK IAM authentication, Kafka access control lists don’t apply to IAM identities, so grant the equivalent permissions in an IAM policy instead.
Configure
The Kafka Consumer integration is enabled by default and monitors each Kafka cluster that you identify with a CXDOT integration annotation or a static target. Each target reports on a whole cluster through any one of its brokers, so configure exactly one target for each cluster. With a target for each broker, the integration reports every time series once per broker. To configure the Kafka Consumer integration, follow these steps:-
To monitor a Kafka cluster running in Kubernetes, add a CXDOT integration annotation
to the Service in front of the brokers, such as a bootstrap Service. The
integrations.cxdot.paloaltonetworks.com/servicekey makes the Service’s cluster IP address the single target for the Kafka cluster, and its value can reference SASL credentials stored in a Kubernetes Secret in the same namespace. Annotate the Service, not the broker pods, because each annotated broker pod becomes a separate target for the whole cluster. For example, add the following annotation to your bootstrap Service:The Collector replaces${target.host}with the Service’s cluster IP address,${target.port.kafka}with the Service port namedkafka, and${k8s_secret:kafka-monitoring/password}with thepasswordkey of thekafka-monitoringSecret. To allow the Collector to read the Secret, add the Service’s namespace toapiServer.metadataServer.secretAllowlistin the Helm values for your CXDOT Collector. -
Optional: To monitor Kafka clusters outside your Kubernetes cluster, configure
static targets. Add one entry for each cluster, and point it at a bootstrap address
that resolves to several brokers, such as a DNS name, a Kubernetes Service, or a load
balancer. For example, add the following to the
values.yamlfile for the CXDOT Collector Helm chart:This integration instance monitors exactly the listed clusters and doesn’t discover annotated targets. In a Kubernetes deployment, one elected replica of the cluster-tier CXDOT Collector monitors static targets. For more information, see architecture. -
Optional: Configure separate integration instances to collect from both annotated
Services and static targets. For example, add the following to the
values.yamlfile for the CXDOT Collector Helm chart:Thekafka_consumer/externalkey defines a named instance for the static targets. Configuring a named instance suppresses the implicit default instance, so the emptykafka_consumer: {}entry is required to keep discovering annotated targets. -
Optional: If your static targets require authentication or TLS, configure the
authandtlssettings, and provide the password through your deployment’s secret management. For example, the following Helm values authenticate withSCRAM-SHA-512, verify the brokers’ certificates against a certificate authority file mounted into the Collector, and reference theKAFKA_CONSUMER_SASL_PASSWORDenvironment variable:Theauthsetting accepts one mechanism at a time:saslwithPLAIN,SCRAM-SHA-256,SCRAM-SHA-512, or MSK IAM authentication, orkerberos. The Collector rejects anauthblock that sets bothsaslandkerberoswhen it loads the configuration. For MSK IAM authentication, set the AWS Region inauth.sasl.aws_msk.region, because the Collector doesn’t detect it. The Collector masks passwords in its debug output and logs. For annotated targets, supply credentials and TLS settings in the CXDOT integration annotation instead, and supply the certificate authority from a Kubernetes Secret withtls.ca_pem: ${k8s_secret:kafka-monitoring/ca.crt}. The Collector rejects an annotation that setstls.ca_file,auth.kerberos.keytab_file, or any other setting that names a file, unless the file is undercollection.cxdot.integrations.discovery.kubernetes.annotation_restrictions.allowed_collector_file_pathsin your Helm values, which allows no files by default. -
Optional: Limit what the integration collects from clusters with many topics,
partitions, or consumer groups. The integration reports several time series for each
partition of every topic that
topic_matchadmits, and requests the offsets of all of those partitions on every collection. For static targets, narrowtopic_matchandgroup_match. For example, the following Helm values report only topics whose names start withorders.and consumer groups whose names start withorders-:To limit collection further, removetopicsfromscrapersto stop collecting topic and partition metrics, or turn off individual metrics withmetrics. A CXDOT integration annotation sets these settings in its own block. -
Optional: Disable the integration. For example, add the following to the
values.yamlfile for the CXDOT Collector Helm chart:
Validate
To validate the Kafka Consumer integration, follow these steps:-
In the
Live Telemetry Analyzer,
add the following filters:
__name__=cxdot.integration.target.healthcxdot.integration.name=kafka_consumer
1. A value of0means the CXDOT Collector hit an error collecting from that cluster. Thecxdot.integration.targetattribute identifies the cluster by the bootstrap address the Collector connects to. -
In
Metrics Explorer,
run the following query:
Confirm that the query returns a time series for each consumer group and each topic where the group has committed offsets, with the group’s total lag across the topic’s partitions as the value. The
kafka.cluster.idattribute identifies the cluster.
Troubleshooting
- The health metric reports
1, but consumer group metrics are missing or intermittent: The health metric needs a response from only one broker, but consumer group metrics need a connection to every broker. When the Collector can’t reach one broker at its advertised address, it drops all consumer group metrics for that collection interval, along with the offsets of the partitions that broker leads. Check the advertised addresses as described in the Prerequisites section. - Consumer group and partition time series appear once for each broker pod: More than one target monitors the same cluster, typically because the annotations are on the broker pods instead of the cluster’s Service. Annotate the Service instead, and list each cluster once among static targets.
- A consumer group reports members but no offsets or lag, or lag is missing for some
partitions: The integration reports offsets and lag only for partitions where the group
has committed an offset. A partition the group hasn’t committed to has no lag time
series, rather than a lag of
0. Lag is the partition’s high watermark minus the group’s committed offset, and is never less than0.
Configuration reference
Configure one Apache Kafka Consumer integration instance with the following settings. In Helm values, place these settings undercollection.cxdot.integrations.kafka_consumer. In a
Collector configuration file, place them under cxdot.integrations.kafka_consumer.
Optional settings
-
enabledType:boolean. Optional. Default:true. Whether to enable this Kafka Consumer integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it. -
endpointsType:array of object. Optional. Default:[]. Kafka clusters to monitor, one entry per cluster. Eachendpointis thehost:portof one broker in that cluster. The Collector learns the remaining brokers from cluster metadata and must be able to reach every broker at the address it advertises. If it can’t reach one of them, it reports no consumer group metrics for that cluster in that collection interval. The Collector first reaches the cluster through this one address, so point eachendpointat a bootstrap address that resolves to several brokers, such as a DNS name, a Kubernetes Service, or a load balancer. An outage of a single broker when the Collector starts then doesn’t leave the cluster unmonitored. Listing several brokers of the same cluster reports that cluster once per entry. When this list is empty or omitted, the integration uses automatic target discovery. When the list contains clusters, this integration instance monitors exactly those clusters and doesn’t automatically discover targets. Configure a second named instance to use both methods. -
endpoints[].endpointType:string. Required.host:portof the cluster’s bootstrap address. -
collection_intervalType:duration. Optional. Default:15s. How often the Collector collects metrics from each Kafka cluster. Targets discovered through annotations don’t use this setting ortimeout. Their annotations determine how often the Collector collects from them. -
timeoutType:duration. Optional. Default:15s. How long one collection from a cluster can run before the Collector abandons it. One collection makes several requests to the brokers, and this setting bounds all of them together. Keep it no longer thancollection_intervalso that a slow collection doesn’t hold up the next one. -
initial_delayType:duration. Optional. Default:1s. How long the Collector waits after starting before it first collects metrics. -
scrapersType:array of string. Optional. Default:["brokers","topics","consumers"]. Minimum items:1. Which groups of metrics to collect.brokersreports the number of brokers in the cluster.topicsreports partition counts, partition offsets, and replica counts. It produces several time series for each partition of every topic thattopic_matchadmits, and the Collector requests the offsets of all of those partitions on every collection.consumersreports the members of each consumer group, and the committed offset and lag of each partition where the group has committed an offset. Lag is the partition’s high watermark, whichkafka.partition.current_offsetalso reports, minus the group’s committed offset, and is never less than0. -
topic_matchType:regex. Optional. Default:^[^_].*$. Regular expression selecting the topics to report, both for partition metrics and for consumer group offsets and lag. The expression uses RE2 syntax and matches any part of a topic name unless you anchor it with^and$. The default excludes topics whose names start with an underscore, which Kafka and common platform components use for internal topics. Withtopicsinscrapers, the Collector reports several time series for each partition of every matched topic and requests all of their offsets on every collection. A narrower expression lowers that cost on clusters with many partitions. Targets discovered through annotations don’t use this setting. Their annotations determine which topics they report. -
group_matchType:regex. Optional. Default:.*. Regular expression selecting the consumer groups to report. The expression uses RE2 syntax and matches any part of a group name unless you anchor it with^and$. The default reports every consumer group in the cluster. This setting doesn’t narrow the partition metrics: withtopicsinscrapers, the Collector still reports several time series for each partition of every topic thattopic_matchadmits and requests all of their offsets on every collection. Targets discovered through annotations don’t use this setting. Their annotations determine which consumer groups they report. -
cluster_aliasType:string. Optional. Default: empty string. A name for the cluster. When set, it is reported as thekafka.cluster.aliasresource attribute on the series the Kafka metrics receiver emits, whateverresource_attributes.cluster_alias.enabledsays. -
client_idType:string. Optional. Default:otel-collector. Minimum length:1. Client ID the Collector presents to the brokers, which they can use for access control lists, quotas, and request logging. -
protocol_versionType:string. Optional. Default: empty string. Kafka version whose protocol the Collector speaks, such as3.8.0. When empty, the Collector negotiates the protocol version with each broker. -
conn_idle_timeoutType:duration. Optional. Default:9m. Roughly how long the Collector keeps an idle connection to a broker open before closing it. -
authType:object. Optional. Authentication for the clusters inendpoints, with either SASL (sasl) or Kerberos (kerberos). When omitted, the Collector connects without authenticating. The Collector rejects the deprecatedplain_textandauth.tlssettings: usesaslwith thePLAINmechanism and the top-leveltlssetting instead. Targets discovered through annotations use the authentication settings in their annotations. -
auth.saslType:object. Optional. SASL authentication settings. -
auth.sasl.mechanismType:string. Optional. SASL mechanism the Collector authenticates with. Set it whenever you configuresasl.PLAIN,SCRAM-SHA-256, andSCRAM-SHA-512authenticate withusernameandpassword.AWS_MSK_IAM_OAUTHBEARERauthenticates to Amazon Managed Streaming for Apache Kafka (MSK) with Identity and Access Management (IAM) credentials from Amazon Web Services (AWS). The Collector loads those credentials from the default AWS credential chain, such as environment variables or a role for its Kubernetes service account, and signs them for the region inaws_msk.region. MSK clients that use IAM authentication connect over TLS, so configuretlsas well. Allowed values:PLAIN,SCRAM-SHA-256,SCRAM-SHA-512,AWS_MSK_IAM_OAUTHBEARER. -
auth.sasl.passwordType:string. Optional. Password for thePLAIN,SCRAM-SHA-256, andSCRAM-SHA-512mechanisms, which require it. The Collector masks this value in its debug output and logs. -
auth.sasl.aws_mskType:object. Optional. Settings for theAWS_MSK_IAM_OAUTHBEARERmechanism. -
auth.sasl.aws_msk.regionType:string. Optional. AWS Region of the MSK cluster, such asus-east-1. Required for theAWS_MSK_IAM_OAUTHBEARERmechanism: the Collector rejects a target whose Region is empty and reports no health metric for it. -
auth.sasl.usernameType:string. Optional. Username for thePLAIN,SCRAM-SHA-256, andSCRAM-SHA-512mechanisms, which require it. -
auth.kerberosType:object. Optional. Kerberos authentication settings. The Collector authenticates with the keytab inkeytab_filewhen that setting isn’t empty, and withpasswordotherwise. -
auth.kerberos.passwordType:string. Optional. Password forusername, used whenkeytab_fileis empty. The Collector masks this value in its debug output and logs. -
auth.kerberos.config_fileType:string. Optional. Path to the Kerberos configuration file, usuallykrb5.conf, that lists the key distribution centers forrealm. -
auth.kerberos.disable_fast_negotiationType:boolean. Optional. Whether to leave Flexible Authentication Secure Tunneling (FAST) negotiation out of the ticket requests that the Collector sends. If true, the Collector leaves it out. Set it to true when your key distribution center rejects that negotiation. If false, the Collector includes it. -
auth.kerberos.keytab_fileType:string. Optional. Path to a keytab file that holds the key forusername. When set, the Collector authenticates with the keytab instead ofpassword. -
auth.kerberos.realmType:string. Optional. Realm ofusername, such asEXAMPLE.COM. -
auth.kerberos.service_nameType:string. Optional. Service name in the brokers’ principals, such askafka. -
auth.kerberos.use_keytabType:boolean. Optional. The Collector accepts this setting but ignores it. Whether the Collector authenticates with a keytab depends only onkeytab_file. -
auth.kerberos.usernameType:string. Optional. Principal name the Collector authenticates as, without the realm. -
tlsType:object. Optional. Transport Layer Security (TLS) settings for the clusters inendpoints. When this block is present, even empty, the Collector connects to the brokers over TLS and verifies their certificates against the system certificate authority pool, unless othertlssettings change that. When omitted, the Collector connects without TLS. Targets discovered through annotations use the TLS settings in their annotations. -
tls.key_pemType:string. Optional. PEM-encoded client private key content used withcert_pemfor mutual TLS authentication. The Collector masks this value in its debug output and logs. -
tls.tpmType:object. Optional. Trusted platform module settings for the TLS private key. -
tls.tpm.authType:string. Optional. Authorization value for the trusted platform module key. -
tls.tpm.owner_authType:string. Optional. Owner authorization value for the trusted platform module. -
tls.tpm.enabledType:boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key. -
tls.tpm.pathType:string. Optional. Path to the trusted platform module device or Unix domain socket, such as/dev/tpm0or/dev/tpmrm0. -
tls.ca_fileType:string. Optional. Path to a PEM-encoded certificate authority (CA) certificate file used to verify the brokers’ certificates. If omitted, the Collector uses the system certificate authority pool. -
tls.ca_pemType:string. Optional. PEM-encoded certificate authority certificate content used to verify the brokers’ certificates. -
tls.cert_fileType:string. Optional. Path to the PEM-encoded client certificate file to present when the brokers require mutual TLS authentication. -
tls.cert_pemType:string. Optional. PEM-encoded client certificate content to present when the brokers require mutual TLS authentication. -
tls.cipher_suitesType:array of string. Optional. TLS cipher suites that the Collector can use, in preference order. If omitted, the Collector uses the secure defaults from the Go TLS library. For supported names, see https://go.dev/src/crypto/tls/cipher_suites.go. -
tls.curve_preferencesType:array of string. Optional. Elliptic curves that the Collector can use for an elliptic curve Diffie-Hellman ephemeral handshake, in preference order. If omitted, the Collector uses the Go TLS library defaults. -
tls.include_system_ca_certs_poolType:boolean. Optional. Whether to add certificates from the system certificate authority pool to the configured certificate authorities. If true, the Collector trusts both sources. If false, the Collector trusts only the configured certificate authorities. -
tls.insecureType:boolean. Optional. Whether to connect without TLS. If true and no certificate authority is configured, the Collector connects to the brokers without TLS. If false, the Collector connects over TLS. -
tls.insecure_skip_verifyType:boolean. Optional. Whether to skip verification of the brokers’ certificates. If true, the connection is encrypted but isn’t authenticated and is vulnerable to man-in-the-middle attacks. If false, the Collector verifies the certificates. -
tls.key_fileType:string. Optional. Path to the PEM-encoded client private key file used withcert_filefor mutual TLS authentication. -
tls.max_versionType:string. Optional. Maximum TLS protocol version the Collector can use. If omitted, the Collector uses the maximum version supported by the Go TLS library. -
tls.min_versionType:string. Optional. Minimum TLS protocol version the Collector can use. If omitted, the minimum version is TLS 1.2. -
tls.reload_intervalType:duration. Optional. How often the Collector reloads certificate and key files. If omitted, the Collector doesn’t reload the files. -
tls.server_name_overrideType:string. Optional. Server name to verify in each broker’s certificate instead of the broker’s hostname. Use this setting when the certificates name a host other than the addresses the brokers advertise. -
metadataType:object. Optional. How the Collector maintains the cluster metadata it uses to route requests to brokers, and how it retries requests that a broker can’t answer yet. Of these settings, onlyfullhas no effect. -
metadata.fullType:boolean. Optional. Default:true. The Collector accepts this setting but ignores it. -
metadata.refresh_intervalType:duration. Optional. Default:10m0s. How long the Collector keeps the cluster metadata it caches to route requests to brokers before it refreshes that metadata. Each collection still reads the current list of brokers, topics, and partitions for its metrics. If omitted, the Collector refreshes the cached metadata every 10 minutes. -
metadata.retryType:object. Optional. How the Collector retries a request that a broker can’t answer yet, such as during a leader election or while the cluster starts. -
metadata.retry.backoffType:duration. Optional. Default:250ms. How long the Collector waits before it first retries a request. Each later retry waits about twice as long as the one before, up to 5 seconds or this value, whichever is longer. The value can’t be negative. If omitted, the Collector waits 250 milliseconds. -
metadata.retry.maxType:integer. Optional. Default:20. Maximum number of times the Collector retries a request that a broker can’t answer yet before it gives up on that request. The value can’t be negative. If omitted, the Collector retries up to 20 times. -
metricsType:object. Optional. Per-metric settings, keyed by metric name. Setenabled: trueto collect a metric that’s off by default, such askafka.topic.replication_factor, orenabled: falseto stop collecting one. Every metric exceptkafka.brokersalso acceptsattributes, the subset of its attributes to keep, andaggregation_strategy, which combines data points that become identical after the other attributes are dropped:sum,avg,min, ormax. For example,kafka.consumer_group.lagwithattributes: [group, topic]reports one time series per consumer group and topic instead of one per partition. Targets discovered through annotations don’t use this setting. -
resource_attributesType:object. Optional. Which cluster identity resource attributes to report on every metric. -
resource_attributes.cluster_idType:object. Optional. Settings for thekafka.cluster.idresource attribute. -
resource_attributes.cluster_id.enabledType:boolean. Optional. Default:true. Whether to report the cluster ID from the cluster’s metadata as thekafka.cluster.idresource attribute. -
resource_attributes.cluster_aliasType:object. Optional. Settings for thekafka.cluster.aliasresource attribute. -
resource_attributes.cluster_alias.enabledType:boolean. Optional. Default:false. Whether to report thekafka.cluster.aliasresource attribute whencluster_aliasis empty. A non-emptycluster_aliasis always reported, so this setting has no effect alongside one.