Supported telemetry types
The Apache Cassandra integration supports these telemetry types:Prerequisites
The Apache Cassandra integration has the following prerequisites:-
Enable remote Java Management Extensions (JMX) on each Cassandra node. By default,
Cassandra accepts JMX connections only from the machine it runs on. Set the
LOCAL_JMXenvironment variable tonobefore Cassandra starts, or setLOCAL_JMX=noincassandra-env.sh. Cassandra then accepts remote JMX connections on port7199and requires authentication for those connections. For more information, see JMX access in the Cassandra documentation. -
Create a JMX user for the CXDOT Collector. Cassandra reads JMX users from
/etc/cassandra/jmxremote.password, with one username and password per line. Make the file readable only by the user that runs Cassandra. -
Give the JMX user for the CXDOT Collector read-only access, since the integration only
reads metrics. Name an access file with the
com.sun.management.jmxremote.access.filesystem property, and list the user in it withreadonlyaccess. -
In production, encrypt JMX connections with Transport Layer Security (TLS). Set the
com.sun.management.jmxremote.sslandcom.sun.management.jmxremote.registry.sslsystem properties totrue, and point thejavax.net.ssl.keyStoresystem property at the file that holds the node’s certificate and private key. For more information, see JMX with SSL in the Cassandra documentation. For example, the following container environment variables enable remote JMX, an access file, and TLS for the official Cassandra image. This configuration expects the access and certificate files at the specified paths and the certificate file’s password in thecassandra-jmx-tlsSecret: -
Allow connections from the CXDOT Collector to the JMX registry and server ports on
each Cassandra node. The JMX registry uses port
7199by default. Cassandra can advertise a different host and port for the JMX server.
Configure
The Apache Cassandra integration is enabled by default. Configure targets either by annotating pods in your Kubernetes cluster or by listing static endpoints in your Helm values. To configure the Apache Cassandra integration, follow these steps:-
To collect from Cassandra pods in Kubernetes, add a CXDOT integration annotation to
the pod template. The annotation key must end with the container name, and its value
can reference JMX credentials stored in a Kubernetes Secret in the same namespace. For
example, add the following annotation to your Cassandra workload:
The Collector replaces
${target.host}with the pod IP address,${target.port.jmx}with the container port namedjmx, and${k8s_secret:cassandra-jmx/password}with thepasswordkey of thecassandra-jmxSecret. To allow the Collector to read the Secret, add the pod’s namespace toapiServer.metadataServer.secretAllowlistin the Helm values for your CXDOT Collector. -
Optional: To collect from Cassandra nodes outside your Kubernetes cluster, configure
static targets. For example, add the following to the Helm values file for your CXDOT
Collector to read the JMX password from the
cassandra-jmxSecret:When you configure static targets, this integration instance collects from exactly those targets instead of discovered targets. In Kubernetes, the cluster Collector that wins leader election collects from static targets. Every Collector loads the same integration settings and rejects ausernamewithout apassword, so set the password variable on both the node and cluster Collectors. Theusernameandpasswordsettings in your Helm values apply to the targets you list there. For an annotated pod, set them in its CXDOT integration annotation. If the JMX server host or port that Cassandra advertises isn’t reachable from the CXDOT Collector, setadvertised_host_overrideoradvertised_port_override. -
Optional: Configure a second integration instance to collect from both annotated pods
and static targets. For example, add the following to the Helm values file for your
CXDOT Collector:
The
cassandra/externalkey defines a named instance for the static targets. The barecassandrakey continues to discover pods. Set theCASSANDRA_JMX_PASSWORDenvironment variable on the node and cluster Collectors, as the previous step shows. -
Optional: If your static Cassandra targets use TLS for JMX, configure the CXDOT
Collector to trust their certificates. For example, add the following to the Helm
values file for your CXDOT Collector:
The
tlsandregistry_tlssettings in your Helm values apply to the targets you list there. Mount the certificate authority file into the cluster Collector, which collects from static targets. To mount the file from a Secret, use theclusterCollector.extraVolumesandclusterCollector.extraVolumeMountsHelm values. For an annotated pod, setregistry_tlsandtlsin its CXDOT integration annotation, and supply the certificate authority from a Kubernetes Secret withtls.ca_pem: ${k8s_secret:cassandra-tls/ca.crt}. The Collector rejects an annotation that setstls.ca_file, or any other setting that names a file, unless the file is undercollection.cxdot.integrations.discovery.kubernetes.annotation_restrictions.allowed_collector_file_pathsin your Helm values, which allows no files by default. The CXDOT Collector verifies each node’s certificate against the host in that target’s endpoint, which for a discovered pod is the pod’s IP address. If your certificates name a different host, setserver_name_overrideto a name that every node’s certificate carries. -
Optional: Disable the integration. For example, add the following to the Helm values
file for your CXDOT Collector:
Validate
To validate the Apache Cassandra integration, follow these steps:-
In Live Telemetry Analyzer,
filter for
__name__=cxdot.integration.target.healthandcxdot.integration.name=cassandra. Confirm thatcxdot.integration.target.healthreports1for each reachable Cassandra node. A value of0means the CXDOT Collector encountered an error while connecting to or collecting from that node. Thecxdot.integration.target,server.address, andserver.portattributes identify the node. -
In Metrics Explorer,
run the following query:
Confirm that the query returns one time series per Cassandra node, each reporting the number of nodes that node sees as up. While every node is up, the value is the number of nodes in your cluster.
Troubleshooting
- Metrics for a new table, or for a thread pool the node hasn’t used yet, are missing:
Cassandra registers a table’s metrics when you create the table, and Cassandra 4.0 and
later create some thread pools when they’re first used. The integration refreshes its
list of JMX beans at each
bean_detection_interval, which defaults to 10 minutes, so the missing time series appear after the next refresh. cassandra.table.memtable.flushedandcassandra.table.compaction.writtenare missing: Cassandra reports these table metrics starting with Cassandra 3.11. A Cassandra 3.0 node doesn’t report them.cassandra.table.memtable.space_wait.duration.p75andcassandra.table.memtable.space_wait.duration.p95always report0, or are missing: Cassandra releases before 5.0 register these percentiles but never record wait values for them, so they report0regardless of the write load. Cassandra 5.0 doesn’t report them.- A query on one
cassandra.message.typevalue finds no dropped messages on some Cassandra nodes: The message types depend on the Cassandra release. On Cassandra 4.0 and later, the integration reports nine message types using names from earlier releases, such asMUTATIONandREAD. It reports the remaining types using the request and response verb names introduced in Cassandra 4.0, such asMUTATION_RSP. Summing across all message types counts all dropped messages for a node in every release.
Configuration reference
Configure one Apache Cassandra integration instance with the following settings. In Helm values, place these settings undercollection.cxdot.integrations.cassandra. In a Collector
configuration file, place them under cxdot.integrations.cassandra.
Optional settings
-
enabledType:boolean. Optional. Default:true. Whether to enable this Cassandra integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it. -
endpointsType:array of object. Optional. Default:[]. Cassandra node Java Management Extensions (JMX) targets. When this list is empty or omitted, the integration uses automatic target discovery. When the list contains targets, this integration instance collects from exactly those targets and doesn’t automatically discover targets. To combine listed targets with automatic target discovery, list them in a second named instance. Each endpoint can be ahost:portRemote Method Invocation (RMI) registry address or aservice:jmx:rmiURL. -
endpoints[].endpointType:string. Required. Cassandra node JMX endpoint as either ahost:portRemote Method Invocation (RMI) registry address or aservice:jmx:rmiURL. -
registry_tlsType:boolean. Optional. Default:false. Whether to use Transport Layer Security (TLS) for the RMI registry connection. If true, the Collector uses TLS, which Cassandra requires when it setscom.sun.management.jmxremote.registry.ssltotrue. If false, the Collector uses an unencrypted registry connection. The connection to the JMX server returned by the registry uses TLS whenever Cassandra requires it, regardless of this setting. For targets that autodiscovery annotations define, the annotation sets this value instead. -
advertised_host_overrideType:string. Optional. Default: empty string. Host the Collector connects to for the JMX server that the RMI registry returns, instead of the host that Cassandra advertises. When empty, the Collector connects to the host in the target’s endpoint, so thejava.rmi.server.hostnamesetting on the Cassandra node doesn’t affect the Collector. -
advertised_port_overrideType:integer. Optional. Default:0. Minimum:0. Maximum:65535. Port the Collector connects to for the JMX server that the RMI registry returns, instead of the port that Cassandra advertises. When set to0, the Collector uses the advertised port. -
usernameType:string. Optional. Username for JMX authentication. Configureusernameandpasswordtogether. For targets that autodiscovery annotations define, the annotation supplies the credentials instead. -
passwordType:string. Optional. Password for JMX authentication. Configureusernameandpasswordtogether. For targets that autodiscovery annotations define, the annotation supplies the credentials instead. -
collection_intervalType:duration. Optional. Default:15s. How often the Collector collects metrics from each Cassandra JMX endpoint. For targets that autodiscovery annotations define, the annotation sets the interval instead. -
bean_detection_intervalType:duration. Optional. Default:10m. How often the Collector refreshes the set of available JMX beans. Cassandra registers beans for tables when they’re created and for some thread pools when they’re first used, so their metrics appear after the next refresh. -
bean_limitType:integer. Optional. Default:35000. Minimum:1. Maximum number of JMX beans the Collector reads from each Cassandra node in one collection. Cassandra registers a set of beans for each table, so the number of beans grows with the number of tables. When a node has more matching beans than this limit, the Collector skips per-table beans first so it can continue reporting node and Java Virtual Machine (JVM) metrics. -
target_systemType:string. Optional. Default:cassandra. Comma-delimited attribute mapping names to apply. The Cassandra mapping is enabled by default, and the JVM mapping is included automatically. Add-jvmto disable JVM metrics when at least one other mapping is configured. For targets that autodiscovery annotations define, the annotation decides whether to collect JVM metrics, and this setting doesn’t apply. -
attribute_mapping_pathsType:array of string. Optional. Default:[]. Directories and files containing custom JMX attribute mappings. The Collector searches these paths before the embedded Cassandra and JVM mappings. -
tlsType:object. Optional. Default:{}. TLS certificate and client authentication settings for RMI registry and server connections. -
tls.ca_fileType:string. Optional. Path to a PEM-encoded certificate authority (CA) certificate file used to verify the JMX server certificate. If omitted, the Collector uses the system certificate authority pool. -
tls.ca_pemType:string. Optional. PEM-encoded certificate authority certificate content used to verify the JMX server certificate. -
tls.cert_fileType:string. Optional. Path to the PEM-encoded client certificate file to present when the JMX server requires mutual TLS authentication. -
tls.cert_pemType:string. Optional. PEM-encoded client certificate content to present when the JMX server requires mutual TLS authentication. -
tls.cipher_suitesType:array of string. Optional. TLS cipher suites that the Collector can use, in preference order. If omitted, the Collector uses secure default cipher suites. For supported names, see https://go.dev/src/crypto/tls/cipher_suites.go. -
tls.curve_preferencesType:array of string. Optional. Elliptic curves that the Collector can use for an elliptic curve Diffie-Hellman ephemeral handshake, in preference order. If omitted, the Collector uses its default curves. -
tls.include_system_ca_certs_poolType:boolean. Optional. Whether to add certificates from the system certificate authority pool to the configured certificate authorities. If true, the Collector trusts both sources. If false, the Collector trusts only the configured certificate authorities. -
tls.insecureType:boolean. Optional. Whether to disable transport security. This setting must be false. Useregistry_tlsto control transport security for the RMI registry. -
tls.insecure_skip_verifyType:boolean. Optional. Whether to skip verification of the JMX server certificate. If true, the connection is encrypted but isn’t authenticated and is vulnerable to man-in-the-middle attacks. If false, the Collector verifies the certificate. -
tls.key_fileType:string. Optional. Path to the PEM-encoded client private key file used withcert_filefor mutual TLS authentication. -
tls.key_pemType:string. Optional. PEM-encoded client private key content used withcert_pemfor mutual TLS authentication. -
tls.max_versionType:string. Optional. Maximum TLS protocol version the Collector can use. If omitted, the Collector uses the highest TLS version it supports. -
tls.min_versionType:string. Optional. Minimum TLS protocol version the Collector can use. If omitted, the minimum version is TLS 1.2. -
tls.reload_intervalType:duration. Optional. How often the Collector reloads certificate and key files. If omitted, the Collector doesn’t reload the files. -
tls.server_name_overrideType:string. Optional. Server name to verify in the JMX server certificate instead of the host in the target’s endpoint. Use this setting when the certificate names a different host, such as for a discovered pod, whose endpoint host is the pod’s IP address. -
tls.tpmType:object. Optional. Trusted Platform Module (TPM) settings for the TLS private key. -
tls.tpm.authType:string. Optional. Authorization value for the TPM key. -
tls.tpm.enabledType:boolean. Optional. Whether to use a TPM for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or an in-memory key. -
tls.tpm.owner_authType:string. Optional. Owner authorization value for the TPM. -
tls.tpm.pathType:string. Optional. Path to the TPM device or Unix domain socket, such as/dev/tpm0or/dev/tpmrm0.