Skip to main content
Cortex XCOR receives Amazon CloudWatch metrics through CloudWatch Metric Streams. Configure Amazon Web Services (AWS) to continuously stream metrics to XCOR through the AWS Management Console or Terraform.

Before you begin

Some AWS services publish metrics that exceed the two-hour ingestion age limit in XCOR by the time CloudWatch delivers them. For example, Amazon S3 daily storage metrics can be more than two hours old when delivered. Before streaming a metric, verify that CloudWatch delivers it within two hours of its timestamp. XCOR can read S3 daily storage metrics from CloudWatch instead: see S3 daily storage metrics. Some values, such as DynamoDB table size and item count, aren’t CloudWatch metrics, and XCOR reads them from the AWS service directly: see DynamoDB table size and item count metrics. Some AWS settings aren’t CloudWatch metrics at all, so CloudWatch Metric Streams can’t send them. XCOR can read each AWS Lambda function’s configured timeout and memory size from your AWS accounts: see Lambda function configuration metrics. Amazon ECS pending task counts aren’t CloudWatch metrics, so CloudWatch Metric Streams can’t deliver them. XCOR can read them from the Amazon ECS API: see Amazon ECS pending tasks. The storage allocated to an Amazon RDS DB instance is a setting, not a CloudWatch metric, so CloudWatch Metric Streams can’t deliver it. XCOR can read it from the Amazon RDS API: see Amazon RDS allocated storage. An Amazon EC2 instance’s system status and launch time aren’t CloudWatch metrics either. XCOR can read them from the Amazon EC2 API: see Amazon EC2 host status and instance age.

Control metric volume

CloudWatch Metric Streams generate metrics that consume Standard Metrics License capacity. Before configuring ingestion, create a drop rule if you want to exclude unwanted metrics. The following Chronoctl YAML resource definition creates a drop rule that drops all metrics from CloudWatch Metric Streams except the AWS/CloudWatch/MetricStreams and AWS/Firehose namespaces required by the validation dashboards.
Chronoctl example
Modify the rule to allow metrics from additional AWS namespaces. The following example allows metrics from the AWS ECS namespace.
Chronoctl example
To reduce the AWS cost of streaming unwanted metrics, configure CloudWatch Metric Streams to include or exclude specific namespaces:
  1. In the AWS Management Console, go to Streams.
  2. Edit the Metric Stream.
  3. Under Metrics to be streamed, include or exclude namespaces.

Configure CloudWatch Metric Streams

In each AWS account and region you want to collect metrics from, a CloudWatch Metric Stream sends data to Amazon Data Firehose. The Firehose stream forwards the data to the CloudWatch ingest endpoint in your XCOR tenant. XCOR processes the metrics and makes them available for queries, monitors, and dashboards.

CloudWatch roles and permissions

To use CloudWatch Metric Streams in XCOR, you must configure a CloudWatch Metric Stream in each AWS account and region. The account you use to set up the CloudWatch Metric Stream must have either the CloudWatchFullAccess policy and iam:PassRole permission or all of the following permissions:
  • iam:PassRole
  • cloudwatch:PutMetricStream
  • cloudwatch:DeleteMetricStream
  • cloudwatch:GetMetricStream
  • cloudwatch:ListMetricStreams
  • cloudwatch:StartMetricStreams
  • cloudwatch:StopMetricStreams
  • iam:CreateRole
  • iam:PutRolePolicy

XCOR authentication

Create or use an API token for an XCOR restricted service account with write-only permission. For more information, see Create a restricted service account. You must also provide your XCOR organization name. This is the <ORG_NAME> segment in https://<ORG_NAME>.chronosphere.io.

AWS resources and IAM roles

The setup creates the following resources and Identity and Access Management (IAM) roles:

AWS resources

  • Amazon S3 bucket: Stores eligible failed deliveries from Amazon Data Firehose.
  • CloudWatch log group: Stores logs from the Firehose stream.
  • Amazon Data Firehose stream: Sends metrics to XCOR through an HTTP endpoint and stores eligible failed deliveries in the S3 bucket.
  • IAM role for Amazon Data Firehose: Grants the Firehose stream the following permissions to access the S3 bucket and CloudWatch Logs:
    • s3:AbortMultipartUpload
    • s3:GetBucketLocation
    • s3:GetObject
    • s3:ListBucket
    • s3:ListBucketMultipartUploads
    • s3:PutObject
    • logs:PutLogEvents

IAM roles

The setup also creates an IAM role with these permissions so CloudWatch Metric Streams can publish data to the Firehose stream:
  • firehose:PutRecord
  • firehose:PutRecordBatch

Apply the configuration

Choose either the Terraform setup or the AWS Management Console setup.
  1. Set values for the following Terraform variables, and then apply the configuration.
    Terraform example

Validate CloudWatch Metric Stream ingestion

After setup, wait for data to arrive in XCOR. Validate ingestion with the operational dashboards in XCOR and the AWS Management Console.
The included dashboards query the default CloudWatch metric names. For tenants configured for Datadog-compatible naming, validate the snake_case metric names in Metrics Explorer.
The integration also includes a Metrics Overview dashboard that shows CloudWatch Metric Streams delivery volume and publish errors by account, region, and stream. The Metrics Overview dashboard is available from the Amazon Web Services integration in the Integrations Hub.

Validate the CloudWatch Metrics Ingestion & Health dashboard

The CloudWatch Metrics Ingestion & Health dashboard displays the health of your CloudWatch Metric Streams integration with XCOR. The CloudWatch Metric Streams and Data Firehose panel groups rely on CloudWatch metrics sent from those services to XCOR. To populate these charts, include metrics from the AWS/Firehose and AWS/CloudWatch/MetricStreams namespaces in your CloudWatch Metric Streams configuration.
  1. In XCOR, go to Dashboards.
  2. In the search bar, enter CloudWatch, and then select the CloudWatch Metrics Ingestion & Health dashboard. The metrics ingestion panel group displays information about the CloudWatch metrics XCOR received.
  3. Use the Data Firehose records received by Amazon Resource Name chart to confirm that the ingestion API received Data Firehose records.
  4. Use the CloudWatch metric updates received chart to confirm the number of CloudWatch metric updates XCOR extracts from the Data Firehose records.
  5. Confirm that the Transformed metrics chart shows no Rejected Data Points.
  6. Use the Unique time series by AWS metric namespace chart to confirm that the stream received by XCOR includes metrics from the AWS namespaces you want.

Validate the Metric Stream dashboard

In the AWS Management Console, use the Metric Updates chart for specific metrics to validate that metrics are streaming.
  1. In the AWS Management Console, go to CloudWatch > Metric Streams.
  2. Select chronosphere-cloudwatch-metric-stream to view the status and operational statistics.
  3. Verify that the Status is Running.
  4. Use the Metric Updates chart to verify how many updates have been sent. If the stream is working, the chart should report a non-zero number of updates.
  5. Verify whether any errors were reported in the Errors chart. The value should be 0.

Validate the Amazon Data Firehose status

In the AWS Management Console, use the Firehose charts to validate that metrics are streaming.
  1. In the AWS Management Console, go to Amazon Data Firehose > Firehose Streams.
  2. Select chronosphere-cloudwatch-firehose to view status and operational statistics for the Firehose stream.
  3. The Incoming bytes, Incoming put requests, and Incoming records charts should all report non-zero values.
  4. The HTTP endpoint delivery success chart should report 100%.
  5. The Records delivered to HTTP endpoint chart should report a non-zero value.

Query metrics about AWS CloudWatch Metric Streams

Query metrics about AWS CloudWatch Metric Streams in XCOR. These metrics don’t appear in XCOR Metrics Explorer if a drop rule excludes all AWS metrics. Modify the rule to allow metrics from at least one namespace, such as all metrics from the AWS/CloudWatch/MetricStreams namespace. For examples, see Control metric volume.
  1. In XCOR, go to Explorers > Metrics Explorer.
  2. In the query box, enter aws_cloudwatch_metricstreams to view a list of AWS metrics received from the CloudWatch Metric Stream.
  3. Select the metric you want to query to add it to the query prompt.
Use the selected metric in a query. For default metric naming, run this query to report the rate of CloudWatch metric updates:
For Datadog-compatible metric naming, run:

Metric semantics

Metric naming conventions

Ingested CloudWatch metric names in XCOR follow this pattern:
  • <namespace>: The namespace is lowercased, and XCOR replaces all forward slash (/) and period (.) characters in the CloudWatch namespace with underscores (_). All AWS service namespaces follow the naming convention AWS/<ServiceName>, where <ServiceName> is replaced with the service name. In XCOR, the ingested metrics therefore begin with aws_<servicename>. For a list of AWS services and their respective namespaces, see AWS services that publish CloudWatch metrics. If you create custom metrics, the namespace you set for the metric becomes the metric name prefix in XCOR.
  • <MetricName>: By default, XCOR preserves the CloudWatch metric name’s case. Tenants configured for Datadog-compatible naming receive the metric name in snake_case.
  • <statistic>: XCOR appends the CloudWatch statistic’s name (count, sum, maximum, minimum, average). If you define additional statistics for a metric, XCOR appends the corresponding CloudWatch metric statistic name (pXX).
For examples, see Example metric names.

Metric labeling conventions

XCOR adds CloudWatch metric dimensions as labels to each time series, following the pattern dimension_<DimensionName>. For examples, see Example metric names.

Stream resource attributes

CloudWatch Metric Streams include OpenTelemetry Protocol (OTLP) resource attributes, which XCOR merges into the time series. XCOR replaces periods (.) with underscores (_) in attribute key names. CloudWatch Metric Streams includes the following resource attributes in the data it sends through Amazon Data Firehose:
  • aws_exporter_arn: The Amazon Resource Name (ARN) of the CloudWatch Metric Stream, which uniquely identifies the metric writer instance.
  • cloud_account_id: The AWS account associated with the CloudWatch Metric Stream, such as 123456789012.
  • cloud_provider: The value is always aws.
  • cloud_region: The AWS region of the Amazon Data Firehose sending the stream, such as us-east-2.
For examples, see Example metric names.

Add custom resource attributes using stream parameters

Define custom key-value pairs as parameters that Amazon Data Firehose includes in each HTTP request. XCOR treats all additional parameters as resource attributes and merges them into the time series.
Your custom parameters take precedence over the default CloudWatch resource attributes. Avoid parameter names that conflict with default CloudWatch attribute names.
Configure additional parameters through Terraform or the AWS Management Console.
To configure parameters using Terraform:
  1. Edit the Terraform module.
  2. In the aws_kinesis_firehose_delivery_stream resource definition, modify the request_configuration block to define additional common_attributes. The AWS Management Console names this setting Parameters, while the API name is common_attributes. The following example configures the content_encoding parameter as GZIP and defines two common_attributes, testname and testname2:
    Terraform example

Example metric names

Given a CloudWatch metric with the following attributes:
  • The namespace AWS/EBS (AWS Service)
  • The metric name VolumeReadBytes
  • The dimension VolumeId
  • The custom Firehose destination parameter environment
  • The CloudWatch metric resource attributes aws_exporter_arn, cloud_account_id, cloud_provider, and cloud_region
The following examples focus on the dimension and resource attributes. Every series also carries Namespace, MetricName, and instance (the metric stream’s ARN) labels. XCOR creates metrics with these names and labels:
Given a CloudWatch metric with the following attributes:
  • The namespace Buildkite (custom metrics)
  • The metric name RunningJobsCount
  • The custom Firehose destination parameter environment
  • No dimensions
  • CloudWatch metric resource attributes aws_exporter_arn, cloud_account_id, cloud_provider, and cloud_region
XCOR creates metrics with these names and labels:

S3 daily storage metrics

Amazon S3 publishes its daily storage metrics, BucketSizeBytes and NumberOfObjects, to CloudWatch once a day, and they don’t arrive in XCOR through CloudWatch Metric Streams. XCOR can read them from CloudWatch in your AWS accounts instead, with the same AWS read role it uses to read your resource tags.
XCOR Support turns on these reads for your tenant. To request them, contact XCOR Support.
XCOR excludes both metrics from the aggregation recommendations for every tenant, because their values arrive after the window aggregation uses.

Prerequisites for S3 daily storage metrics

XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-only cloudwatch:ListMetrics and cloudwatch:GetMetricData actions these reads use.
  • AWS bills GetMetricData requests to your account per metric requested. XCOR requests each bucket’s metrics once or twice a day.
  • S3 publishes a bucket’s storage metrics only in the bucket’s own region. XCOR reads no buckets in a region it doesn’t read your resource tags in.

S3 daily storage metric names

Each time series has these labels:
  • bucketname: The bucket’s name.
  • storagetype: The storage class, such as StandardStorage or GlacierStorage. For aws_s3_number_of_objects_average, it’s always AllStorageTypes.
  • Namespace (always AWS/S3), cloud_provider (always aws), cloud_account_id and cloud_region.
  • aws_account_alias: The alias XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowed iam:ListAccountAliases, which read role template 1.2.0 grants.
  • The bucket’s resource tags, where XCOR reads them.
These time series don’t have dimension_* labels or the stream resource attributes that streamed metrics have.

S3 daily storage timestamps

XCOR writes each UTC day’s value with a timestamp one hour after the day ends. For example, the value for November 16 has the timestamp November 17, 01:00 UTC. CloudWatch timestamps the same value at the start of its day, so the CloudWatch console shows it a day earlier than XCOR does. XCOR reads each day once or twice between about 01:05 and 03:00 UTC the next day. After XCOR Support turns on these reads, the first value arrives between about 01:05 and 02:05 UTC the next day, for the day that was in progress when they were turned on. If they’re turned on before about 03:00 UTC, XCOR might also write the value for the day before. It doesn’t write values for earlier days. If AWS publishes a day’s value later than about 01:55 UTC the next day, the day might be missing. If AWS changes a day’s value after that, XCOR keeps the value it read.

Query S3 daily storage metrics

Each value has one data point a day, so queries need a lookback of more than a day:
  • To show each day’s value at the end of its day in a time series panel, and to line it up with other metrics, add offset -1h to the selector:
  • To show the latest value in a stat panel, a table, or a monitor, use last_over_time over 27 hours, with no offset. A day’s value can arrive up to about two hours after its timestamp, so a window of one day returns no data each day from 01:00 UTC until that day’s value arrives. An instant query without last_over_time returns no data for most of each day.

Validate S3 daily storage ingestion

After 03:10 UTC on the day after XCOR Support turns on these reads, run this query. It returns the number of buckets with a value in the last 27 hours:

DynamoDB table size and item count metrics

Amazon DynamoDB reports each table’s size and item count through its DescribeTable API. These values aren’t CloudWatch metrics, so CloudWatch Metric Streams can’t deliver them. XCOR can read them from DynamoDB in your AWS accounts instead, with the same AWS read role it uses to read your resource tags.
XCOR Support turns on these reads for your tenant. To request them, contact XCOR Support.

Prerequisites for DynamoDB table metrics

XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-only dynamodb:ListTables and dynamodb:DescribeTable actions these reads use.
  • In each account and region, XCOR lists your tables with ListTables and calls DescribeTable for each table it lists, up to 10,000 tables. If a call fails, for example because AWS still throttles it after retries, the tables after it get no value from that read.
  • XCOR writes no values for a table whose resource policy denies the read role dynamodb:DescribeTable.

DynamoDB table metric names

Both metrics are gauges. Each time series has these labels:
  • dimension_TableName: The table’s name. Streamed AWS/DynamoDB metrics that have a TableName dimension carry the same label.
  • Namespace (always AWS/DynamoDB), cloud_provider (always aws), cloud_account_id and cloud_region.
  • aws_account_alias: The alias configured for the account in XCOR, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowed iam:ListAccountAliases, which read role template 1.2.0 grants. It can also be absent for a while when XCOR can’t read the AWS account alias, and appears once a read succeeds. The label is part of each time series’ identity, so an account with no alias configured gets new time series when its AWS account alias appears, changes, or goes away.
  • The table’s resource tags, where XCOR reads them.
These time series don’t have the stream resource attributes that streamed metrics have. XCOR writes no values for a table that’s being archived or is archived, or for a table whose AWS KMS key DynamoDB can’t access (INACCESSIBLE_ENCRYPTION_CREDENTIALS). For an inaccessible key, values resume once DynamoDB can access the key again.

DynamoDB table metric timestamps

XCOR reads your tables every minute by default, and timestamps each value at the time it reads it. XCOR Support can set a longer interval, up to four minutes. DynamoDB updates TableSizeBytes and ItemCount approximately every six hours, so each time series holds the same value for up to six hours and then changes in a single step. Recent writes to a table don’t appear in these metrics until DynamoDB updates the values, and a new table can read 0 bytes and 0 items until DynamoDB first updates them.

Query DynamoDB table metrics

Each table’s values change only when DynamoDB updates them, so compare values over periods longer than six hours:
  • To show the ten largest tables:
  • To show how much each table grew over the last day:
  • To show the total items in each account:

Validate DynamoDB table metric ingestion

After XCOR Support turns on these reads, run this query. It returns the number of tables with a value in the last 10 minutes:

Lambda function configuration metrics

Each AWS Lambda function’s configured timeout and memory size are settings, not CloudWatch metrics, so CloudWatch Metric Streams can’t send them. XCOR can read them from your AWS accounts with the Lambda ListFunctions API, using the same AWS read role it uses to read your resource tags, and write them as two metrics, aws_lambda_timeout and aws_lambda_memorysize. Dashboards and monitors converted from Datadog that query aws.lambda.timeout and aws.lambda.memorysize read these two metrics with no change.
XCOR Support turns on these reads for your tenant. To request them, contact XCOR Support.
XCOR reads each function’s unpublished ($LATEST) configuration, for up to 10,000 functions in each account and region. It doesn’t read published versions or aliases. From each function, XCOR copies its name, timeout, memory size and architecture. ListFunctions also returns each function’s environment variables, which XCOR doesn’t read, log, store or write.

Prerequisites for Lambda function configuration metrics

XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. There’s nothing else to configure for Lambda. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-only lambda:ListFunctions action these reads use. An account whose role is older denies the action, and XCOR writes no Lambda metrics for it.

Lambda function configuration metric names

Both metrics are gauges. XCOR writes the values as Lambda returns them, with no unit conversion: Lambda’s “MB” is a mebibyte. A function whose configuration omits one of the values has no sample of that metric until a later read returns it. The names are the same for tenants configured for Datadog-compatible naming. Each time series has these labels:
  • dimension_FunctionName: The function’s name, such as orders-api.
  • architecture: The function’s instruction set architecture, x86_64 or arm64. It’s absent if Lambda returns other than one architecture for the function.
  • Namespace (always AWS/Lambda), cloud_provider (always aws), cloud_account_id and cloud_region.
  • aws_account_alias: The alias XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowed iam:ListAccountAliases, which read role template 1.2.0 grants. The label is part of each time series’ identity, so an account with no alias set by XCOR Support gets new time series when its AWS account alias appears, changes or goes away.
  • xcor.integration.name: Always aws. The name contains dots, so quote it in a selector: {"xcor.integration.name"="aws"}.
  • The function’s resource tags, where XCOR reads them.
These time series don’t have the dimension_Resource or dimension_ExecutedVersion labels that some streamed Lambda metrics have, because XCOR reads only each function’s unpublished configuration. They don’t have the stream resource attributes that streamed metrics have.

Lambda function configuration timestamps

XCOR reads every function in each account and region once a minute by default, and writes one sample of each metric per function. Each sample’s timestamp is the time XCOR finished reading the function’s account and region. A change to a function’s configuration appears in XCOR at the next read.

Query Lambda function configuration metrics

Each function has a sample about once a minute, so instant queries and range queries work as they do for streamed metrics:
Functions in different accounts or regions can share a name. To keep them apart, add cloud_account_id and cloud_region to the labels you aggregate by. The timeout is in seconds, and the streamed Lambda Duration metric is in milliseconds. To compare one function’s longest run with its timeout, multiply the timeout by 1,000:
Tenants configured for Datadog-compatible naming receive the streamed metric as aws_lambda_duration_maximum.

Validate Lambda function configuration ingestion

A few minutes after XCOR Support turns on these reads, run this query. It returns the number of functions with a timeout value in the last 10 minutes:

Troubleshoot Lambda function configuration metrics

  • No Lambda metrics for one account. Check the account’s read role version. A role older than 1.1.0 denies lambda:ListFunctions.
  • Gaps during busy periods. ListFunctions shares Lambda’s control-plane request quota of 15 requests a second in each region with your other Lambda management calls. When Lambda throttles a read, XCOR writes the functions it read before the throttle and reads all of them again at the next read.
  • A function has no architecture label. Lambda returned other than one architecture for it. Its two metrics are still written.
  • Some functions in a large region have no metrics. XCOR reads at most 10,000 functions in each account and region. In a region with more, the functions after the first 10,000 that Lambda lists get no metrics.

Amazon ECS pending tasks

XCOR can read two counts of Amazon Elastic Container Service (Amazon ECS) tasks in the PENDING state from the ECS API in your AWS accounts, with the same AWS read role it uses to read your resource tags. Neither count is in the AWS/ECS CloudWatch namespace, so CloudWatch Metric Streams can’t deliver them. CloudWatch Container Insights, where you turn it on, publishes its own per-service PendingTaskCount, which is a separate metric.
XCOR Support turns on these reads for your tenant. To request them, contact XCOR Support.
For each account and region, XCOR lists your clusters with ListClusters. In each cluster, it lists the container instances with ListContainerInstances and describes them with DescribeContainerInstances, and lists the services with ListServices and describes them with DescribeServices.

Prerequisites for Amazon ECS pending tasks

XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-only actions these reads use:
  • ecs:ListClusters
  • ecs:ListContainerInstances
  • ecs:DescribeContainerInstances
  • ecs:ListServices
  • ecs:DescribeServices
Amazon ECS limits the rate of API calls in each account and region, and the limit is shared with your own deployments and tools. Each XCOR read of an account and region makes at most five ECS calls a second, a quarter or less of each limit’s refill rate.

Amazon ECS pending tasks metric names

Both metrics are gauges. The _sum in aws_ecs_service_pending_sum is part of the name, which matches dashboards converted from Datadog’s aws.ecs.service.pending. The value is the service’s current count, not a sum over time. Each time series has these labels:
  • dimension_ClusterName: The cluster’s name.
  • dimension_InstanceId: On aws_ecs_pending_tasks_count only. The container instance’s Amazon EC2 instance ID, such as i-0123456789abcdef0. For an external instance registered with ECS Anywhere, the managed instance ID, such as mi-0123456789abcdef0.
  • dimension_ServiceName: On aws_ecs_service_pending_sum only. The service’s name.
  • Namespace (always AWS/ECS), cloud_provider (always aws), cloud_account_id and cloud_region.
  • xcor.integration.name: Always aws.
  • aws_account_alias: The alias XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowed iam:ListAccountAliases, which read role template 1.2.0 grants. A permissions boundary or service control policy on the role must also allow it. It can also be absent for a while when XCOR can’t read the AWS account alias, for example while IAM throttles the request. The label is part of each time series’ identity, so an account with no alias set by XCOR Support gets new time series when its AWS account alias appears, changes, or goes away.
  • Resource tags, where XCOR reads them: the service’s tags on aws_ecs_service_pending_sum, or the cluster’s tags for a service that has none, and the cluster’s tags on aws_ecs_pending_tasks_count. A tag never replaces a label the time series already has.
These time series don’t have the MetricName, instance or aws_exporter_arn labels that streamed metrics have, or the custom resource attributes you add with stream parameters. The example drop rule in Control metric volume filters on aws_exporter_arn, so it doesn’t drop them. XCOR writes no value for a service that’s being deleted (DRAINING) or is deleted (INACTIVE), for a container instance that’s deregistered (INACTIVE), or for a container instance whose description has no ec2InstanceId. When ECS can’t describe a container instance or a service and reports it in the response’s failures list, XCOR writes no value for it on that read. When a container instance is deregistered or a service is deleted, its time series stops.

Amazon ECS pending tasks timestamps

XCOR reads every cluster every four minutes by default. Each value is the count when XCOR read it, with the timestamp of the time XCOR finished reading that account and region, which can be minutes later in a large account.

Query Amazon ECS pending tasks metrics

Each time series has one data point per read, so read a range longer than the read interval, such as 10 minutes, with last_over_time. For example, this query returns the pending tasks in each cluster’s services:
A container instance runs tasks from any service, so aws_ecs_pending_tasks_count has no dimension_ServiceName label. A selector such as dimension_ServiceName=~".*" still matches it, but one that names a service doesn’t. The xcor.integration.name label and your AWS resource tag labels (aws.tag.<key>) contain dots, so quote them in a selector:

Validate Amazon ECS pending tasks ingestion

After XCOR Support turns on these reads, allow up to about 20 minutes for the first values to arrive.
  1. In XCOR, go to Explorers > Metrics Explorer.
  2. Run this query to count the services read in each account and region:
    For container instances, run:
    The queries read 10 minutes so that a single missed read doesn’t hide a resource. Each account and region with ECS services, or with container instances, should appear. Each resource’s newest value should be no older than one read interval, four minutes by default, plus the time XCOR takes to read all your accounts and regions, which can be several minutes when you have many.
If an account or region is missing, check that its read role allows the five ECS actions in Prerequisites for Amazon ECS pending tasks.

Amazon RDS allocated storage

XCOR can read the storage allocated to each Amazon Relational Database Service (Amazon RDS) DB instance from the RDS API in your AWS accounts, with the same AWS read role it uses to read your resource tags, and write it as the aws_rds_total_storage_space metric. The allocated storage is a setting in the instance’s description, not a CloudWatch metric, so CloudWatch Metric Streams can’t deliver it.
XCOR Support turns on these reads for your tenant. To request them, contact XCOR Support.
For each account and region, XCOR lists your DB instances with DescribeDBInstances, reading every page of results, and reads each instance’s AllocatedStorage setting. If a request for a later page fails, the instances on the pages before it still get values from that read.

Prerequisites for Amazon RDS allocated storage

XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-only rds:DescribeDBInstances action these reads use.
  • If your accounts run template 1.0.0, update the stack or StackSet to 1.1.0 or later. The update changes the role’s policy in place, and the role’s Amazon Resource Name (ARN) doesn’t change.
  • If a permissions boundary or service control policy applies to the role, it must also allow rds:DescribeDBInstances. AWS doesn’t report an error at deploy time when a boundary blocks an action.
XCOR reads each region of an account separately. Where the role isn’t allowed the action, AWS refuses the read, and XCOR writes no RDS allocated storage for that account and region. A service control policy with a region condition can refuse the read in some of an account’s regions and allow it in others. Accounts and regions whose roles allow it aren’t affected.

Amazon RDS allocated storage metric names

The metric is a gauge, with one time series for each DB instance. It has no statistic suffix, and the name is the same for tenants configured for Datadog-compatible naming. Each time series has these labels:
  • dimension_DBInstanceIdentifier: The DB instance identifier. Streamed per-instance AWS/RDS metrics carry the same label.
  • dimension_EngineName: The database engine, such as postgres or mysql.
  • dimension_DatabaseClass: The DB instance class, such as db.r6g.large.
  • dimension_DBClusterIdentifier: The Multi-AZ DB cluster the instance belongs to. It’s absent if the instance isn’t a member of one.
  • name: The DB instance identifier, the same value as dimension_DBInstanceIdentifier.
  • dbname: The instance’s initial database name. It’s absent if none was set.
  • Namespace (always AWS/RDS), cloud_provider (always aws), cloud_account_id and cloud_region.
  • xcor.integration.name: Always aws.
  • aws_account_alias: The alias XCOR Support set for the account, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowed iam:ListAccountAliases, which read role template 1.2.0 grants. A permissions boundary or service control policy on the role must also allow it. It can also be absent for a while when XCOR can’t read the AWS account alias, for example while IAM throttles the request. The label is part of each time series’ identity, so an account with no alias set by XCOR Support gets new time series when its AWS account alias appears, changes, or goes away.
  • The instance’s resource tags, where XCOR reads them. A tag never replaces a label the time series already has, such as a tag whose label name is name. If adding an instance’s tags would exceed the label limits for a time series, XCOR writes that time series without any of its tags. If XCOR can’t read your resource tags in time for a read, that read’s values arrive without tags. Where XCOR Support turns on RDS attributes for your resource tags, these also include the instance’s engine, its dbclusteridentifier, and its Availability Zone as dimension_AvailabilityZone.
Every label is part of a time series’ identity, so a change to an instance’s class or tags, or a Multi-AZ failover that moves it to another Availability Zone, starts a new time series. The old one stays in queries that read a range including its last value. To count each instance once, aggregate by dimension_DBInstanceIdentifier with max, as the queries in this section do. The dimension_ labels have the names streamed RDS metrics use, but they aren’t combined the same way. CloudWatch publishes an instance’s streamed RDS metrics with dimension_DBInstanceIdentifier only, and publishes engine-wide and class-wide aggregates as separate time series. Each aws_rds_total_storage_space time series carries the instance’s engine, class and cluster together. These time series don’t have the MetricName, instance or aws_exporter_arn labels that streamed metrics have, or the custom resource attributes you add with stream parameters. The example drop rule in Control metric volume filters on aws_exporter_arn, so it doesn’t drop them. XCOR writes no value for an Amazon Aurora DB instance. An Aurora instance stores its data in its cluster’s shared storage volume, and AWS documents AllocatedStorage as not applying to Aurora instances in CreateDBInstance, so its value isn’t the instance’s storage. aws_rds_total_storage_space is never an Aurora cluster’s volume size. To monitor Aurora storage, use the cluster’s VolumeBytesUsed metric, which CloudWatch Metric Streams deliver when your stream includes the AWS/RDS namespace. For details, see Amazon CloudWatch metrics for Amazon Aurora. XCOR also writes no value for Amazon Neptune and Amazon DocumentDB instances, which DescribeDBInstances also returns, or for an instance that DescribeDBInstances returns without an AllocatedStorage value. When an instance is deleted, its time series stops.

Amazon RDS allocated storage timestamps

XCOR reads every DB instance every two minutes by default. XCOR Support can set an interval from one to four minutes. Each value is the instance’s allocated storage when XCOR read it, with the timestamp of the time XCOR finished reading that account and region. A change to an instance’s allocated storage appears in XCOR at the first read after DescribeDBInstances returns the new value.

Query Amazon RDS allocated storage

Each time series has one data point per read, and its newest value can be as old as one read interval plus the time XCOR takes to read all your accounts and regions. Read a range longer than that, such as 10 minutes, with last_over_time, rather than an instant query, which can return nothing for some instances when XCOR takes several minutes to read your accounts. DB instances in different accounts or regions can share an identifier, so keep cloud_account_id and cloud_region in the labels you aggregate by. For example, this query returns the storage allocated to all your DB instances in each account and region:
A query that combines a streamed metric with aws_rds_total_storage_space must aggregate each side to the labels both have first, because the streamed metric has labels this one doesn’t. A division of the raw time series matches nothing. For example, this query returns the percentage of each DB instance’s allocated storage that’s free, from the streamed FreeStorageSpace metric, which is also in bytes, with default metric naming:
For Datadog-compatible metric naming, run:
The xcor.integration.name label and your AWS resource tag labels (aws.tag.<key>) contain dots, so quote them in a selector:

Validate Amazon RDS allocated storage ingestion

After XCOR Support turns on these reads, allow up to about 20 minutes for the first values to arrive.
  1. In XCOR, go to Explorers > Metrics Explorer.
  2. Run this query to count the DB instances read in each account and region:
    The query reads 10 minutes so that a single missed read doesn’t hide an instance. Each account and region with RDS DB instances other than Aurora, Neptune and DocumentDB instances should appear. Each instance’s newest value should be no older than one read interval, two minutes by default, plus the time XCOR takes to read all your accounts and regions, which can be several minutes when you have many.
If an account or region is missing, check that its read role allows rds:DescribeDBInstances, as Prerequisites for Amazon RDS allocated storage describes.

Amazon EC2 host status and instance age

Each running Amazon EC2 instance’s system status and launch time are fields of the instance’s description, not CloudWatch metrics, so CloudWatch Metric Streams can’t send them. XCOR can read them from your AWS accounts with the EC2 DescribeInstanceStatus and DescribeInstances APIs, using the same AWS read role it uses to read your resource tags, and write them as two metrics, aws_ec2_host_ok and aws_ec2_instance_age. Dashboards and monitors converted from Datadog that query aws.ec2.instance_age read aws_ec2_instance_age. Converted queries for aws.ec2.host_ok read the streamed aws_ec2_status_check_failed_average metric instead, so they don’t read aws_ec2_host_ok. That’s the streamed metric’s name for tenants configured for Datadog-compatible naming. Other tenants receive it as aws_ec2_StatusCheckFailed_average.
XCOR Support turns on these reads for your tenant. To request them, contact XCOR Support.
In each account and region, XCOR lists every running instance with DescribeInstances and reads its system status with DescribeInstanceStatus. Instances that are pending, stopping, stopped, shutting down or terminated aren’t listed, so their metrics stop until the instance runs again. Instances that AWS services manage for you are included.

Prerequisites for Amazon EC2 host status and instance age

XCOR reads the AWS accounts and regions where it reads your AWS resource tags directly from AWS. There’s nothing else to configure for EC2. Each account needs the XCOR AWS read role at template version 1.1.0 or later, which grants the read-only actions these reads use:
  • ec2:DescribeInstances
  • ec2:DescribeInstanceStatus
An account whose role is older denies both actions, and XCOR writes no EC2 host metrics for it.

Amazon EC2 host status and instance age metric names

Both metrics are gauges. aws_ec2_host_ok follows only the system status: a failing instance status check doesn’t change it, and a new instance reads 1 while its status is initializing. The age is measured from the instance’s last launch, so a stop and start resets it. An instance whose description has no launch time has no aws_ec2_instance_age sample from that read. Each time series has these labels:
  • dimension_InstanceId: The instance ID, such as i-0123456789abcdef0.
  • dimension_InstanceType: The instance type, such as m5.large. It’s absent from an aws_ec2_host_ok sample for an instance that started during the read.
  • dimension_AvailabilityZone: The instance’s Availability Zone, such as us-east-1a.
  • Namespace (always AWS/EC2), cloud_provider (always aws), cloud_account_id and cloud_region.
  • xcor.integration.name: Always aws.
  • aws_account_alias: The alias configured for the account in the XCOR AWS integration, or else its AWS account alias. It’s absent if the account has neither, or if the account’s read role isn’t allowed iam:ListAccountAliases, which read role template 1.2.0 grants. A permissions boundary or service control policy on the role must also allow it. It can also be absent for a while when XCOR can’t read the AWS account alias, for example while IAM fails or throttles the request, and appears once a read succeeds. The label is part of each time series’ identity, so an account with no alias configured gets new time series when its AWS account alias appears, changes, or goes away.
  • The instance’s resource tags, where XCOR reads them, such as aws.tag.name from the Name tag.
These time series don’t have a host or dimension_AutoScalingGroupName label. They don’t have the MetricName, instance or aws_exporter_arn labels that streamed metrics have, or the custom resource attributes you add with stream parameters. The example drop rule in Control metric volume filters on aws_exporter_arn, so it doesn’t drop them.

Amazon EC2 host status and instance age timestamps

XCOR reads every running instance in each account and region every two minutes by default, and writes one sample of each metric per instance. Each sample’s timestamp is the time XCOR finished reading that account and region. XCOR Support can set an interval from one to four minutes. A change EC2 reports in an instance’s system status appears in XCOR at the next read.

Query Amazon EC2 host status and instance age metrics

Each instance has a sample every read, so at the default interval instant queries and range queries work as they do for streamed metrics. For example, this query returns the instances whose system status is impaired:
This query returns the instances that have run for more than 90 days since their last launch, with each one’s age in days:
The metrics have no host label, so a monitor converted from Datadog that groups by host puts every matching instance in one group and alerts on their average age. Group by dimension_InstanceId for one alert per instance. The xcor.integration.name label and your AWS resource tag labels (aws.tag.<key>) contain dots, so quote them in a selector:

Validate Amazon EC2 host status and instance age ingestion

After XCOR Support turns on these reads, run this query. It counts the running instances with a host status value in the last 10 minutes in each account and region:
For instance age, run:
The queries read 10 minutes so that a single missed read doesn’t hide an instance. Each account and region with running instances should appear. Each instance’s newest value should be no older than one read interval, two minutes by default, plus the time XCOR takes to read all your accounts and regions.

Troubleshoot Amazon EC2 host status and instance age metrics

  • No EC2 host metrics for one account. Check the account’s read role version. A role older than 1.1.0 denies ec2:DescribeInstances and ec2:DescribeInstanceStatus.
  • aws_ec2_instance_age but no aws_ec2_host_ok for an account. The role grants ec2:DescribeInstances but not ec2:DescribeInstanceStatus.
  • Gaps during busy periods. EC2 limits each API action’s request rate in each account and region, and the calls your own tools make count against the same limit. When EC2 throttles a read, XCOR writes what it read before the throttle and reads every instance again at the next read.
  • aws_ec2_host_ok is 1 while an instance status check fails. Only an impaired system status reads 0. A new instance reads 1 while its status is initializing, and a failing instance status check doesn’t change the metric.
  • An instance’s age dropped to near 0. The age is measured from the instance’s last launch, which a stop and start resets.
  • An aws_ec2_host_ok time series has no dimension_InstanceType label. The instance started during that read. The next read writes the label.

Monitor drop rules

To view the rate at which the example CloudWatch Metric Streams drop rule drops data points, use the following PromQL query: