Before you begin
Some AWS services publish metrics that exceed the two-hour ingestion age limit in XCOR by the time CloudWatch delivers them. For example, Amazon S3 daily storage metrics can be more than two hours old when delivered. Before streaming a metric, verify that CloudWatch delivers it within two hours of its timestamp. XCOR can read S3 daily storage metrics from CloudWatch instead: see S3 daily storage metrics. Some values, such as DynamoDB table size and item count, aren’t CloudWatch metrics, and XCOR reads them from the AWS service directly: see DynamoDB table size and item count metrics. Some AWS settings aren’t CloudWatch metrics at all, so CloudWatch Metric Streams can’t send them. XCOR can read each AWS Lambda function’s configured timeout and memory size from your AWS accounts: see Lambda function configuration metrics. Amazon ECS pending task counts aren’t CloudWatch metrics, so CloudWatch Metric Streams can’t deliver them. XCOR can read them from the Amazon ECS API: see Amazon ECS pending tasks. The storage allocated to an Amazon RDS DB instance is a setting, not a CloudWatch metric, so CloudWatch Metric Streams can’t deliver it. XCOR can read it from the Amazon RDS API: see Amazon RDS allocated storage. An Amazon EC2 instance’s system status and launch time aren’t CloudWatch metrics either. XCOR can read them from the Amazon EC2 API: see Amazon EC2 host status and instance age.Control metric volume
CloudWatch Metric Streams generate metrics that consume Standard Metrics License capacity. Before configuring ingestion, create a drop rule if you want to exclude unwanted metrics. The following Chronoctl YAML resource definition creates a drop rule that drops all metrics from CloudWatch Metric Streams except theAWS/CloudWatch/MetricStreams and AWS/Firehose namespaces required by
the validation dashboards.
Chronoctl example
Chronoctl example
- In the AWS Management Console, go to Streams.
- Edit the Metric Stream.
- Under Metrics to be streamed, include or exclude namespaces.
Configure CloudWatch Metric Streams
In each AWS account and region you want to collect metrics from, a CloudWatch Metric Stream sends data to Amazon Data Firehose. The Firehose stream forwards the data to the CloudWatch ingest endpoint in your XCOR tenant. XCOR processes the metrics and makes them available for queries, monitors, and dashboards.CloudWatch roles and permissions
To use CloudWatch Metric Streams in XCOR, you must configure a CloudWatch Metric Stream in each AWS account and region. The account you use to set up the CloudWatch Metric Stream must have either theCloudWatchFullAccess policy
and iam:PassRole permission or all of the following permissions:
iam:PassRolecloudwatch:PutMetricStreamcloudwatch:DeleteMetricStreamcloudwatch:GetMetricStreamcloudwatch:ListMetricStreamscloudwatch:StartMetricStreamscloudwatch:StopMetricStreamsiam:CreateRoleiam:PutRolePolicy
XCOR authentication
Create or use an API token for an XCOR restricted service account with write-only permission. For more information, see Create a restricted service account. You must also provide your XCOR organization name. This is the<ORG_NAME>
segment in https://<ORG_NAME>.chronosphere.io.
AWS resources and IAM roles
The setup creates the following resources and Identity and Access Management (IAM) roles:AWS resources
- Amazon S3 bucket: Stores eligible failed deliveries from Amazon Data Firehose.
- CloudWatch log group: Stores logs from the Firehose stream.
- Amazon Data Firehose stream: Sends metrics to XCOR through an HTTP endpoint and stores eligible failed deliveries in the S3 bucket.
- IAM role for Amazon Data Firehose: Grants the Firehose stream the following
permissions to access the S3 bucket and CloudWatch Logs:
s3:AbortMultipartUploads3:GetBucketLocations3:GetObjects3:ListBuckets3:ListBucketMultipartUploadss3:PutObjectlogs:PutLogEvents
IAM roles
The setup also creates an IAM role with these permissions so CloudWatch Metric Streams can publish data to the Firehose stream:firehose:PutRecordfirehose:PutRecordBatch
Apply the configuration
Choose either the Terraform setup or the AWS Management Console setup.- Terraform
- AWS Management Console
-
Set values for the following
Terraform
variables, and then apply the configuration.
Terraform example
Validate CloudWatch Metric Stream ingestion
After setup, wait for data to arrive in XCOR. Validate ingestion with the operational dashboards in XCOR and the AWS Management Console.The included dashboards query the default CloudWatch metric names. For tenants
configured for Datadog-compatible naming, validate the
snake_case metric names in
Metrics Explorer.Validate the CloudWatch Metrics Ingestion & Health dashboard
The CloudWatch Metrics Ingestion & Health dashboard displays the health of your CloudWatch Metric Streams integration with XCOR. The CloudWatch Metric Streams and Data Firehose panel groups rely on CloudWatch metrics sent from those services to XCOR. To populate these charts, include metrics from theAWS/Firehose and
AWS/CloudWatch/MetricStreams namespaces in your CloudWatch Metric Streams
configuration.
- In XCOR, go to Dashboards.
- In the search bar, enter CloudWatch, and then select the CloudWatch Metrics Ingestion & Health dashboard. The metrics ingestion panel group displays information about the CloudWatch metrics XCOR received.
- Use the Data Firehose records received by Amazon Resource Name chart to confirm that the ingestion API received Data Firehose records.
- Use the CloudWatch metric updates received chart to confirm the number of CloudWatch metric updates XCOR extracts from the Data Firehose records.
- Confirm that the Transformed metrics chart shows no Rejected Data Points.
- Use the Unique time series by AWS metric namespace chart to confirm that the stream received by XCOR includes metrics from the AWS namespaces you want.
Validate the Metric Stream dashboard
In the AWS Management Console, use the Metric Updates chart for specific metrics to validate that metrics are streaming.- In the AWS Management Console, go to CloudWatch > Metric Streams.
- Select
chronosphere-cloudwatch-metric-streamto view the status and operational statistics. - Verify that the Status is
Running. - Use the Metric Updates chart to verify how many updates have been sent. If the stream is working, the chart should report a non-zero number of updates.
- Verify whether any errors were reported in the Errors chart. The value should be 0.
Validate the Amazon Data Firehose status
In the AWS Management Console, use the Firehose charts to validate that metrics are streaming.- In the AWS Management Console, go to Amazon Data Firehose > Firehose Streams.
- Select
chronosphere-cloudwatch-firehoseto view status and operational statistics for the Firehose stream. - The Incoming bytes, Incoming put requests, and Incoming records charts should all report non-zero values.
- The HTTP endpoint delivery success chart should report 100%.
- The Records delivered to HTTP endpoint chart should report a non-zero value.
Query metrics about AWS CloudWatch Metric Streams
Query metrics about AWS CloudWatch Metric Streams in XCOR. These metrics don’t appear in XCOR Metrics Explorer if a drop rule excludes all AWS metrics. Modify the rule to allow metrics from at least one namespace, such as all metrics from theAWS/CloudWatch/MetricStreams namespace. For examples, see
Control metric volume.
- In XCOR, go to Explorers > Metrics Explorer.
- In the query box, enter
aws_cloudwatch_metricstreamsto view a list of AWS metrics received from the CloudWatch Metric Stream. - Select the metric you want to query to add it to the query prompt.
Metric semantics
Metric naming conventions
Ingested CloudWatch metric names in XCOR follow this pattern:-
<namespace>: The namespace is lowercased, and XCOR replaces all forward slash (/) and period (.) characters in the CloudWatch namespace with underscores (_). All AWS service namespaces follow the naming conventionAWS/<ServiceName>, where<ServiceName>is replaced with the service name. In XCOR, the ingested metrics therefore begin withaws_<servicename>. For a list of AWS services and their respective namespaces, see AWS services that publish CloudWatch metrics. If you create custom metrics, the namespace you set for the metric becomes the metric name prefix in XCOR. -
<MetricName>: By default, XCOR preserves the CloudWatch metric name’s case. Tenants configured for Datadog-compatible naming receive the metric name insnake_case. -
<statistic>: XCOR appends the CloudWatch statistic’s name (count,sum,maximum,minimum,average). If you define additional statistics for a metric, XCOR appends the corresponding CloudWatch metric statistic name (pXX).
Metric labeling conventions
XCOR adds CloudWatch metric dimensions as labels to each time series, following the patterndimension_<DimensionName>. For examples, see
Example metric names.
Stream resource attributes
CloudWatch Metric Streams include OpenTelemetry Protocol (OTLP) resource attributes, which XCOR merges into the time series. XCOR replaces periods (.) with underscores (_) in attribute key names.
CloudWatch Metric Streams includes the following resource attributes in the data it
sends through Amazon Data Firehose:
aws_exporter_arn: The Amazon Resource Name (ARN) of the CloudWatch Metric Stream, which uniquely identifies the metric writer instance.cloud_account_id: The AWS account associated with the CloudWatch Metric Stream, such as123456789012.cloud_provider: The value is alwaysaws.cloud_region: The AWS region of the Amazon Data Firehose sending the stream, such asus-east-2.
Add custom resource attributes using stream parameters
Define custom key-value pairs as parameters that Amazon Data Firehose includes in each HTTP request. XCOR treats all additional parameters as resource attributes and merges them into the time series.Your custom parameters take precedence over the default CloudWatch resource
attributes. Avoid parameter names that conflict with default CloudWatch attribute
names.
- Terraform
- AWS Management Console
To configure parameters using Terraform:
- Edit the Terraform module.
-
In the
aws_kinesis_firehose_delivery_streamresource definition, modify therequest_configurationblock to define additionalcommon_attributes. The AWS Management Console names this setting Parameters, while the API name iscommon_attributes. The following example configures thecontent_encodingparameter asGZIPand defines twocommon_attributes,testnameandtestname2:Terraform example
Example metric names
Given a CloudWatch metric with the following attributes:- The namespace
AWS/EBS(AWS Service) - The metric name
VolumeReadBytes - The dimension
VolumeId - The custom Firehose destination parameter
environment - The CloudWatch metric resource attributes
aws_exporter_arn,cloud_account_id,cloud_provider, andcloud_region
Namespace, MetricName, and instance (the metric stream’s ARN)
labels.
XCOR creates metrics with these names and labels:
- The namespace
Buildkite(custom metrics) - The metric name
RunningJobsCount - The custom Firehose destination parameter
environment - No dimensions
- CloudWatch metric resource attributes
aws_exporter_arn,cloud_account_id,cloud_provider, andcloud_region
S3 daily storage metrics
Amazon S3 publishes its daily storage metrics,BucketSizeBytes and
NumberOfObjects, to CloudWatch once a day, and they don’t arrive in
XCOR through CloudWatch Metric Streams. XCOR can read them from
CloudWatch in your AWS accounts instead, with the same AWS read role it uses to
read your resource tags.
XCOR Support turns on these reads for your tenant. To request them, contact
XCOR Support.
Prerequisites for S3 daily storage metrics
XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-onlycloudwatch:ListMetrics and cloudwatch:GetMetricData actions these reads use.
- AWS bills
GetMetricDatarequests to your account per metric requested. XCOR requests each bucket’s metrics once or twice a day. - S3 publishes a bucket’s storage metrics only in the bucket’s own region. XCOR reads no buckets in a region it doesn’t read your resource tags in.
S3 daily storage metric names
Each time series has these labels:
bucketname: The bucket’s name.storagetype: The storage class, such asStandardStorageorGlacierStorage. Foraws_s3_number_of_objects_average, it’s alwaysAllStorageTypes.Namespace(alwaysAWS/S3),cloud_provider(alwaysaws),cloud_account_idandcloud_region.aws_account_alias: The alias XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowediam:ListAccountAliases, which read role template 1.2.0 grants.- The bucket’s resource tags, where XCOR reads them.
dimension_* labels or the stream resource
attributes that streamed metrics have.
S3 daily storage timestamps
XCOR writes each UTC day’s value with a timestamp one hour after the day ends. For example, the value for November 16 has the timestamp November 17, 01:00 UTC. CloudWatch timestamps the same value at the start of its day, so the CloudWatch console shows it a day earlier than XCOR does. XCOR reads each day once or twice between about 01:05 and 03:00 UTC the next day. After XCOR Support turns on these reads, the first value arrives between about 01:05 and 02:05 UTC the next day, for the day that was in progress when they were turned on. If they’re turned on before about 03:00 UTC, XCOR might also write the value for the day before. It doesn’t write values for earlier days. If AWS publishes a day’s value later than about 01:55 UTC the next day, the day might be missing. If AWS changes a day’s value after that, XCOR keeps the value it read.Query S3 daily storage metrics
Each value has one data point a day, so queries need a lookback of more than a day:-
To show each day’s value at the end of its day in a time series panel, and to
line it up with other metrics, add
offset -1hto the selector: -
To show the latest value in a stat panel, a table, or a monitor, use
last_over_timeover 27 hours, with no offset. A day’s value can arrive up to about two hours after its timestamp, so a window of one day returns no data each day from 01:00 UTC until that day’s value arrives. An instant query withoutlast_over_timereturns no data for most of each day.
Validate S3 daily storage ingestion
After 03:10 UTC on the day after XCOR Support turns on these reads, run this query. It returns the number of buckets with a value in the last 27 hours:DynamoDB table size and item count metrics
Amazon DynamoDB reports each table’s size and item count through itsDescribeTable API. These values aren’t CloudWatch metrics, so CloudWatch
Metric Streams can’t deliver them. XCOR can read them from DynamoDB in your AWS
accounts instead, with the same AWS read role it uses to read your resource
tags.
XCOR Support turns on these reads for your tenant. To request them, contact
XCOR Support.
Prerequisites for DynamoDB table metrics
XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-onlydynamodb:ListTables and dynamodb:DescribeTable actions these reads use.
- In each account and region, XCOR lists your tables with
ListTablesand callsDescribeTablefor each table it lists, up to 10,000 tables. If a call fails, for example because AWS still throttles it after retries, the tables after it get no value from that read. - XCOR writes no values for a table whose resource policy denies the read role
dynamodb:DescribeTable.
DynamoDB table metric names
Both metrics are gauges. Each time series has these labels:
dimension_TableName: The table’s name. StreamedAWS/DynamoDBmetrics that have aTableNamedimension carry the same label.Namespace(alwaysAWS/DynamoDB),cloud_provider(alwaysaws),cloud_account_idandcloud_region.aws_account_alias: The alias configured for the account in XCOR, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowediam:ListAccountAliases, which read role template 1.2.0 grants. It can also be absent for a while when XCOR can’t read the AWS account alias, and appears once a read succeeds. The label is part of each time series’ identity, so an account with no alias configured gets new time series when its AWS account alias appears, changes, or goes away.- The table’s resource tags, where XCOR reads them.
INACCESSIBLE_ENCRYPTION_CREDENTIALS). For an inaccessible key, values
resume once DynamoDB can access the key again.
DynamoDB table metric timestamps
XCOR reads your tables every minute by default, and timestamps each value at the time it reads it. XCOR Support can set a longer interval, up to four minutes. DynamoDB updatesTableSizeBytes and ItemCount approximately every six
hours, so each time series holds the same value for up to six hours and then
changes in a single step. Recent writes to a table don’t appear in these
metrics until DynamoDB updates the values, and a new table can read 0 bytes
and 0 items until DynamoDB first updates them.
Query DynamoDB table metrics
Each table’s values change only when DynamoDB updates them, so compare values over periods longer than six hours:-
To show the ten largest tables:
-
To show how much each table grew over the last day:
-
To show the total items in each account:
Validate DynamoDB table metric ingestion
After XCOR Support turns on these reads, run this query. It returns the number of tables with a value in the last 10 minutes:Lambda function configuration metrics
Each AWS Lambda function’s configured timeout and memory size are settings, not CloudWatch metrics, so CloudWatch Metric Streams can’t send them. XCOR can read them from your AWS accounts with the LambdaListFunctions API, using the
same AWS read role it uses to read your resource tags, and write them as two
metrics, aws_lambda_timeout and aws_lambda_memorysize.
Dashboards and monitors converted from Datadog that query aws.lambda.timeout
and aws.lambda.memorysize read these two metrics with no change.
XCOR Support turns on these reads for your tenant. To request them, contact
XCOR Support.
$LATEST) configuration, for up to
10,000 functions in each account and region. It doesn’t read published versions
or aliases. From each function, XCOR copies its name,
timeout, memory size and architecture. ListFunctions also returns each
function’s environment variables, which XCOR doesn’t read, log, store or
write.
Prerequisites for Lambda function configuration metrics
XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. There’s nothing else to configure for Lambda. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-onlylambda:ListFunctions action these reads use. An account
whose role is older denies the action, and XCOR writes no Lambda metrics for
it.
Lambda function configuration metric names
Both metrics are gauges. XCOR writes the values as Lambda returns them, with no
unit conversion: Lambda’s “MB” is a mebibyte. A function whose configuration
omits one of the values has no sample of that metric until a later read returns
it. The names are the same for tenants configured for Datadog-compatible naming.
Each time series has these labels:
dimension_FunctionName: The function’s name, such asorders-api.architecture: The function’s instruction set architecture,x86_64orarm64. It’s absent if Lambda returns other than one architecture for the function.Namespace(alwaysAWS/Lambda),cloud_provider(alwaysaws),cloud_account_idandcloud_region.aws_account_alias: The alias XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowediam:ListAccountAliases, which read role template 1.2.0 grants. The label is part of each time series’ identity, so an account with no alias set by XCOR Support gets new time series when its AWS account alias appears, changes or goes away.xcor.integration.name: Alwaysaws. The name contains dots, so quote it in a selector:{"xcor.integration.name"="aws"}.- The function’s resource tags, where XCOR reads them.
dimension_Resource or
dimension_ExecutedVersion labels that some streamed Lambda metrics have,
because XCOR reads only each function’s unpublished configuration. They don’t
have the stream resource attributes that streamed metrics have.
Lambda function configuration timestamps
XCOR reads every function in each account and region once a minute by default, and writes one sample of each metric per function. Each sample’s timestamp is the time XCOR finished reading the function’s account and region. A change to a function’s configuration appears in XCOR at the next read.Query Lambda function configuration metrics
Each function has a sample about once a minute, so instant queries and range queries work as they do for streamed metrics:cloud_account_id and cloud_region to the labels you aggregate
by.
The timeout is in seconds, and the streamed Lambda Duration metric is in
milliseconds. To compare one function’s longest run with its timeout, multiply
the timeout by 1,000:
aws_lambda_duration_maximum.
Validate Lambda function configuration ingestion
A few minutes after XCOR Support turns on these reads, run this query. It returns the number of functions with a timeout value in the last 10 minutes:Troubleshoot Lambda function configuration metrics
- No Lambda metrics for one account. Check the account’s read role version.
A role older than 1.1.0 denies
lambda:ListFunctions. - Gaps during busy periods.
ListFunctionsshares Lambda’s control-plane request quota of 15 requests a second in each region with your other Lambda management calls. When Lambda throttles a read, XCOR writes the functions it read before the throttle and reads all of them again at the next read. - A function has no
architecturelabel. Lambda returned other than one architecture for it. Its two metrics are still written. - Some functions in a large region have no metrics. XCOR reads at most 10,000 functions in each account and region. In a region with more, the functions after the first 10,000 that Lambda lists get no metrics.
Amazon ECS pending tasks
XCOR can read two counts of Amazon Elastic Container Service (Amazon ECS) tasks in thePENDING state from the ECS API in your AWS accounts, with the
same AWS read role it uses to read your resource tags. Neither count is in the
AWS/ECS CloudWatch namespace, so CloudWatch Metric Streams can’t deliver
them. CloudWatch Container Insights, where you turn it on, publishes its own
per-service PendingTaskCount, which is a separate metric.
XCOR Support turns on these reads for your tenant. To request them, contact
XCOR Support.
ListClusters. In
each cluster, it lists the container instances with ListContainerInstances
and describes them with DescribeContainerInstances, and lists the services
with ListServices and describes them with DescribeServices.
Prerequisites for Amazon ECS pending tasks
XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-only actions these reads use:ecs:ListClustersecs:ListContainerInstancesecs:DescribeContainerInstancesecs:ListServicesecs:DescribeServices
Amazon ECS pending tasks metric names
Both metrics are gauges. The
_sum in aws_ecs_service_pending_sum is part
of the name, which matches dashboards converted from Datadog’s
aws.ecs.service.pending. The value is the service’s current count, not a sum
over time.
Each time series has these labels:
dimension_ClusterName: The cluster’s name.dimension_InstanceId: Onaws_ecs_pending_tasks_countonly. The container instance’s Amazon EC2 instance ID, such asi-0123456789abcdef0. For an external instance registered with ECS Anywhere, the managed instance ID, such asmi-0123456789abcdef0.dimension_ServiceName: Onaws_ecs_service_pending_sumonly. The service’s name.Namespace(alwaysAWS/ECS),cloud_provider(alwaysaws),cloud_account_idandcloud_region.xcor.integration.name: Alwaysaws.aws_account_alias: The alias XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowediam:ListAccountAliases, which read role template 1.2.0 grants. A permissions boundary or service control policy on the role must also allow it. It can also be absent for a while when XCOR can’t read the AWS account alias, for example while IAM throttles the request. The label is part of each time series’ identity, so an account with no alias set by XCOR Support gets new time series when its AWS account alias appears, changes, or goes away.- Resource tags, where XCOR reads them: the service’s tags on
aws_ecs_service_pending_sum, or the cluster’s tags for a service that has none, and the cluster’s tags onaws_ecs_pending_tasks_count. A tag never replaces a label the time series already has.
MetricName, instance or
aws_exporter_arn labels that streamed metrics have, or the custom resource
attributes you add with stream parameters. The example drop rule in
Control metric volume filters on aws_exporter_arn,
so it doesn’t drop them.
XCOR writes no value for a service that’s being deleted (DRAINING) or is
deleted (INACTIVE), for a container instance that’s deregistered
(INACTIVE), or for a container instance whose description has no
ec2InstanceId. When ECS can’t describe a container instance or a service and
reports it in the response’s failures list, XCOR writes no value for it on
that read. When a container instance is deregistered or a service is deleted,
its time series stops.
Amazon ECS pending tasks timestamps
XCOR reads every cluster every four minutes by default. Each value is the count when XCOR read it, with the timestamp of the time XCOR finished reading that account and region, which can be minutes later in a large account.Query Amazon ECS pending tasks metrics
Each time series has one data point per read, so read a range longer than the read interval, such as 10 minutes, withlast_over_time. For example, this
query returns the pending tasks in each cluster’s services:
aws_ecs_pending_tasks_count has no dimension_ServiceName label. A selector
such as dimension_ServiceName=~".*" still matches it, but one that names a
service doesn’t.
The xcor.integration.name label and your AWS resource tag labels
(aws.tag.<key>) contain dots, so quote them in a selector:
Validate Amazon ECS pending tasks ingestion
After XCOR Support turns on these reads, allow up to about 20 minutes for the first values to arrive.- In XCOR, go to Explorers > Metrics Explorer.
-
Run this query to count the services read in each account and region:
For container instances, run:The queries read 10 minutes so that a single missed read doesn’t hide a resource. Each account and region with ECS services, or with container instances, should appear. Each resource’s newest value should be no older than one read interval, four minutes by default, plus the time XCOR takes to read all your accounts and regions, which can be several minutes when you have many.
Amazon RDS allocated storage
XCOR can read the storage allocated to each Amazon Relational Database Service (Amazon RDS) DB instance from the RDS API in your AWS accounts, with the same AWS read role it uses to read your resource tags, and write it as theaws_rds_total_storage_space metric. The allocated storage is a setting in
the instance’s description, not a CloudWatch metric, so CloudWatch Metric
Streams can’t deliver it.
XCOR Support turns on these reads for your tenant. To request them, contact
XCOR Support.
DescribeDBInstances,
reading every page of results, and reads each instance’s AllocatedStorage
setting. If a request for a later page fails, the instances on the pages before
it still get values from that read.
Prerequisites for Amazon RDS allocated storage
XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-onlyrds:DescribeDBInstances action these reads use.
- If your accounts run template 1.0.0, update the stack or StackSet to 1.1.0 or later. The update changes the role’s policy in place, and the role’s Amazon Resource Name (ARN) doesn’t change.
- If a permissions boundary or service control policy applies to the role, it
must also allow
rds:DescribeDBInstances. AWS doesn’t report an error at deploy time when a boundary blocks an action.
Amazon RDS allocated storage metric names
The metric is a gauge, with one time series for each DB instance. It has no
statistic suffix, and the name is the same for tenants configured for
Datadog-compatible naming.
Each time series has these labels:
dimension_DBInstanceIdentifier: The DB instance identifier. Streamed per-instanceAWS/RDSmetrics carry the same label.dimension_EngineName: The database engine, such aspostgresormysql.dimension_DatabaseClass: The DB instance class, such asdb.r6g.large.dimension_DBClusterIdentifier: The Multi-AZ DB cluster the instance belongs to. It’s absent if the instance isn’t a member of one.name: The DB instance identifier, the same value asdimension_DBInstanceIdentifier.dbname: The instance’s initial database name. It’s absent if none was set.Namespace(alwaysAWS/RDS),cloud_provider(alwaysaws),cloud_account_idandcloud_region.xcor.integration.name: Alwaysaws.aws_account_alias: The alias XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowediam:ListAccountAliases, which read role template 1.2.0 grants. A permissions boundary or service control policy on the role must also allow it. It can also be absent for a while when XCOR can’t read the AWS account alias, for example while IAM throttles the request. The label is part of each time series’ identity, so an account with no alias set by XCOR Support gets new time series when its AWS account alias appears, changes, or goes away.- The instance’s resource tags, where XCOR reads them. A tag never replaces a
label the time series already has, such as a tag whose label name is
name. If adding an instance’s tags would exceed the label limits for a time series, XCOR writes that time series without any of its tags. If XCOR can’t read your resource tags in time for a read, that read’s values arrive without tags. Where XCOR Support turns on RDS attributes for your resource tags, these also include the instance’sengine, itsdbclusteridentifier, and its Availability Zone asdimension_AvailabilityZone.
dimension_DBInstanceIdentifier with max, as the queries in this section do.
The dimension_ labels have the names streamed RDS metrics use, but they
aren’t combined the same way. CloudWatch publishes an instance’s streamed RDS
metrics with dimension_DBInstanceIdentifier only, and publishes engine-wide
and class-wide aggregates as separate time series. Each
aws_rds_total_storage_space time series carries the instance’s engine, class
and cluster together.
These time series don’t have the MetricName, instance or
aws_exporter_arn labels that streamed metrics have, or the custom resource
attributes you add with stream parameters. The example drop rule in
Control metric volume filters on aws_exporter_arn,
so it doesn’t drop them.
XCOR writes no value for an Amazon Aurora DB instance. An Aurora instance
stores its data in its cluster’s shared storage volume, and AWS documents
AllocatedStorage as not applying to Aurora instances in
CreateDBInstance,
so its value isn’t the instance’s storage. aws_rds_total_storage_space is
never an Aurora cluster’s volume size. To monitor Aurora storage, use the
cluster’s VolumeBytesUsed metric, which CloudWatch Metric Streams deliver
when your stream includes the AWS/RDS namespace. For details, see
Amazon CloudWatch metrics for Amazon Aurora.
XCOR also writes no value for Amazon Neptune and Amazon DocumentDB instances,
which DescribeDBInstances also returns, or for an instance that
DescribeDBInstances returns without an AllocatedStorage value. When an
instance is deleted, its time series stops.
Amazon RDS allocated storage timestamps
XCOR reads every DB instance every two minutes by default. XCOR Support can set an interval from one to four minutes. Each value is the instance’s allocated storage when XCOR read it, with the timestamp of the time XCOR finished reading that account and region. A change to an instance’s allocated storage appears in XCOR at the first read afterDescribeDBInstances returns
the new value.
Query Amazon RDS allocated storage
Each time series has one data point per read, and its newest value can be as old as one read interval plus the time XCOR takes to read all your accounts and regions. Read a range longer than that, such as 10 minutes, withlast_over_time, rather than an instant query, which can return nothing for
some instances when XCOR takes several minutes to read your accounts.
DB instances in different accounts or regions can share an identifier, so keep
cloud_account_id and cloud_region in the labels you aggregate by. For
example, this query returns the storage allocated to all your DB instances in
each account and region:
aws_rds_total_storage_space
must aggregate each side to the labels both have first, because the streamed
metric has labels this one doesn’t. A division of the raw time series matches
nothing. For example, this query returns the percentage of each DB instance’s
allocated storage that’s free, from the streamed FreeStorageSpace metric,
which is also in bytes, with default metric naming:
xcor.integration.name label and your AWS resource tag labels
(aws.tag.<key>) contain dots, so quote them in a selector:
Validate Amazon RDS allocated storage ingestion
After XCOR Support turns on these reads, allow up to about 20 minutes for the first values to arrive.- In XCOR, go to Explorers > Metrics Explorer.
-
Run this query to count the DB instances read in each account and region:
The query reads 10 minutes so that a single missed read doesn’t hide an instance. Each account and region with RDS DB instances other than Aurora, Neptune and DocumentDB instances should appear. Each instance’s newest value should be no older than one read interval, two minutes by default, plus the time XCOR takes to read all your accounts and regions, which can be several minutes when you have many.
rds:DescribeDBInstances, as
Prerequisites for Amazon RDS allocated storage
describes.
Amazon EC2 host status and instance age
Each running Amazon EC2 instance’s system status and launch time are fields of the instance’s description, not CloudWatch metrics, so CloudWatch Metric Streams can’t send them. XCOR can read them from your AWS accounts with the EC2DescribeInstanceStatus and DescribeInstances APIs, using the same AWS read
role it uses to read your resource tags, and write them as two metrics,
aws_ec2_host_ok and aws_ec2_instance_age.
Dashboards and monitors converted from Datadog that query
aws.ec2.instance_age read aws_ec2_instance_age. Converted queries for
aws.ec2.host_ok read the streamed aws_ec2_status_check_failed_average
metric instead, so they don’t read aws_ec2_host_ok. That’s the streamed
metric’s name for tenants configured for Datadog-compatible naming. Other
tenants receive it as aws_ec2_StatusCheckFailed_average.
XCOR Support turns on these reads for your tenant. To request them, contact
XCOR Support.
DescribeInstances and reads its system status with DescribeInstanceStatus.
Instances that are pending, stopping, stopped, shutting down or terminated
aren’t listed, so their metrics stop until the instance runs again. Instances
that AWS services manage for you are included.
Prerequisites for Amazon EC2 host status and instance age
XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. There’s nothing else to configure for EC2. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-only actions these reads use:ec2:DescribeInstancesec2:DescribeInstanceStatus
Amazon EC2 host status and instance age metric names
Both metrics are gauges.
aws_ec2_host_ok follows only the system status: a
failing instance status check doesn’t change it, and a new instance reads 1
while its status is initializing. The age is measured from the instance’s last
launch, so a stop and start resets it. An instance whose description has no
launch time has no aws_ec2_instance_age sample from that read.
Each time series has these labels:
dimension_InstanceId: The instance ID, such asi-0123456789abcdef0.dimension_InstanceType: The instance type, such asm5.large. It’s absent from anaws_ec2_host_oksample for an instance that started during the read.dimension_AvailabilityZone: The instance’s Availability Zone, such asus-east-1a.Namespace(alwaysAWS/EC2),cloud_provider(alwaysaws),cloud_account_idandcloud_region.xcor.integration.name: Alwaysaws.aws_account_alias: The alias configured for the account in the XCOR AWS integration, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowediam:ListAccountAliases, which read role template 1.2.0 grants. A permissions boundary or service control policy on the role must also allow it. It can also be absent for a while when XCOR can’t read the AWS account alias, for example while IAM fails or throttles the request, and appears once a read succeeds. The label is part of each time series’ identity, so an account with no alias configured gets new time series when its AWS account alias appears, changes, or goes away.- The instance’s resource tags, where XCOR reads them, such as
aws.tag.namefrom theNametag.
host or dimension_AutoScalingGroupName
label. They don’t have the MetricName, instance or aws_exporter_arn
labels that streamed metrics have, or the custom resource attributes you add
with stream parameters. The example drop rule in
Control metric volume filters on aws_exporter_arn,
so it doesn’t drop them.
Amazon EC2 host status and instance age timestamps
XCOR reads every running instance in each account and region every two minutes by default, and writes one sample of each metric per instance. Each sample’s timestamp is the time XCOR finished reading that account and region. XCOR Support can set an interval from one to four minutes. A change EC2 reports in an instance’s system status appears in XCOR at the next read.Query Amazon EC2 host status and instance age metrics
Each instance has a sample every read, so at the default interval instant queries and range queries work as they do for streamed metrics. For example, this query returns the instances whose system status is impaired:host label, so a monitor converted from Datadog that
groups by host puts every matching instance in one group and alerts on their
average age. Group by dimension_InstanceId for one alert per instance.
The xcor.integration.name label and your AWS resource tag labels
(aws.tag.<key>) contain dots, so quote them in a selector:
Validate Amazon EC2 host status and instance age ingestion
After XCOR Support turns on these reads, run this query. It counts the running instances with a host status value in the last 10 minutes in each account and region:Troubleshoot Amazon EC2 host status and instance age metrics
- No EC2 host metrics for one account. Check the account’s read role
version. A role older than 1.1.0 denies
ec2:DescribeInstancesandec2:DescribeInstanceStatus. aws_ec2_instance_agebut noaws_ec2_host_okfor an account. The role grantsec2:DescribeInstancesbut notec2:DescribeInstanceStatus.- Gaps during busy periods. EC2 limits each API action’s request rate in each account and region, and the calls your own tools make count against the same limit. When EC2 throttles a read, XCOR writes what it read before the throttle and reads every instance again at the next read.
aws_ec2_host_okis 1 while an instance status check fails. Only animpairedsystem status reads 0. A new instance reads 1 while its status isinitializing, and a failing instance status check doesn’t change the metric.- An instance’s age dropped to near 0. The age is measured from the instance’s last launch, which a stop and start resets.
- An
aws_ec2_host_oktime series has nodimension_InstanceTypelabel. The instance started during that read. The next read writes the label.