> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingest AWS CloudWatch metrics

> Learn how to send AWS CloudWatch metrics to Cortex XCOR.

Cortex XCOR receives Amazon CloudWatch metrics through
[CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html).
Configure Amazon Web Services (AWS) to continuously stream metrics to XCOR
through the AWS Management Console or
[Terraform](https://docs-xcor.paloaltonetworks.com/tooling/infrastructure/terraform).

## Before you begin

Some AWS services publish metrics that exceed the two-hour ingestion age limit in
XCOR by the time CloudWatch delivers them. For example, Amazon S3 daily
storage metrics can be more than two hours old when delivered. Before streaming a
metric, verify that CloudWatch delivers it within two hours of its timestamp.
XCOR can read S3 daily storage metrics from CloudWatch instead: see
[S3 daily storage metrics](#s3-daily-storage-metrics). Some values, such as
DynamoDB table size and item count, aren't CloudWatch metrics, and XCOR reads
them from the AWS service directly: see
[DynamoDB table size and item count metrics](#dynamodb-table-size-and-item-count-metrics).

Some AWS settings aren't CloudWatch metrics at all, so CloudWatch Metric Streams
can't send them. XCOR can read each AWS Lambda function's configured timeout
and memory size from your AWS accounts: see
[Lambda function configuration metrics](#lambda-function-configuration-metrics).

Amazon ECS pending task counts aren't CloudWatch metrics, so CloudWatch Metric
Streams can't deliver them. XCOR can read them from the Amazon ECS API: see
[Amazon ECS pending tasks](#amazon-ecs-pending-tasks).

The storage allocated to an Amazon RDS DB instance is a setting, not a
CloudWatch metric, so CloudWatch Metric Streams can't deliver it. XCOR can read
it from the Amazon RDS API: see
[Amazon RDS allocated storage](#amazon-rds-allocated-storage).

An Amazon EC2 instance's system status and launch time aren't CloudWatch
metrics either. XCOR can read them from the Amazon EC2 API: see
[Amazon EC2 host status and instance age](#amazon-ec2-host-status-and-instance-age).

### Control metric volume

CloudWatch Metric Streams generate metrics that consume Standard Metrics License
capacity. Before configuring ingestion, create a
[drop rule](https://docs-xcor.paloaltonetworks.com/control/shaping/shape-metrics/rules/drop-rules.md)
if you want to exclude unwanted metrics.

The following
[Chronoctl](https://docs-xcor.paloaltonetworks.com/tooling/chronoctl) YAML resource
definition creates a drop rule that drops all metrics from CloudWatch Metric Streams
except the `AWS/CloudWatch/MetricStreams` and `AWS/Firehose` namespaces required by
the validation dashboards.

```yaml Chronoctl example icon="square-terminal" theme={null}
api_version: v1/config
kind: DropRule
spec:
  slug: drop-cloudwatch-metric-stream-metrics
  name: Drop CloudWatch Metric Stream metrics
  mode: ENABLED
  filters:
    - name: aws_exporter_arn
      value_glob: arn:aws:cloudwatch:*
    - name: Namespace!
      value_glob: '{AWS/CloudWatch/MetricStreams,AWS/Firehose}'
```

Modify the rule to allow metrics from additional AWS namespaces. The following
example allows metrics from the AWS ECS namespace.

```yaml Chronoctl example icon="square-terminal" theme={null}
api_version: v1/config
kind: DropRule
spec:
  slug: drop-cloudwatch-metric-stream-metrics
  name: Drop CloudWatch Metric Stream metrics
  mode: ENABLED
  filters:
    - name: aws_exporter_arn
      value_glob: arn:aws:cloudwatch:*
    - name: Namespace!
      value_glob: '{AWS/CloudWatch/MetricStreams,AWS/Firehose,AWS/ECS}'
```

To reduce the AWS cost of streaming unwanted metrics, configure CloudWatch Metric
Streams to include or exclude specific namespaces:

1. In the AWS Management Console, go to **Streams**.
2. Edit the Metric Stream.
3. Under **Metrics to be streamed**, include or exclude namespaces.

## Configure CloudWatch Metric Streams

In each AWS account and region you want to collect metrics from, a CloudWatch Metric
Stream sends data to Amazon Data Firehose. The Firehose stream forwards the data to
the CloudWatch ingest endpoint in your XCOR tenant.

XCOR processes the metrics and makes them available for queries, monitors,
and dashboards.

```mermaid actions={false} theme={null}
graph LR
  accTitle: AWS CloudWatch metrics delivery
  accDescr: CloudWatch Metric Streams in AWS regions send metrics through Amazon Data Firehose to the Cortex XCOR CloudWatch ingest API.
  subgraph AWS-account
    subgraph US-EAST-2
      cms1[CloudWatch
          Metrics Stream]
      adf1[AWS Data
          Firehose]
    end
    subgraph US-WEST-2
      cms2[CloudWatch
          Metrics Stream]
      adf2[AWS Data
          Firehose]
    end
      cms1 --> adf1
      cms2 --> adf2
  end
  subgraph op[Cortex XCOR]
    subgraph Tenant
      adf3[AWS Data
          Firehose
          ingest API]
      ccp[Cortex XCOR Control
          Plane]
    end
  end
    adf3 --> ccp
    adf1 & adf2 --> adf3

classDef greenFill stroke:#60E684,fill:#2FBF71,stroke-width:2px,color:#FFFFFF
class adf3,ccp greenFill;

classDef greenFillDark stroke:#60E684,fill:#2FBF71,stroke-width:2px,color:#FFFFFF
class op greenFillDark;

classDef blueFill fill:#000243,stroke-width:2px,color:#FFFFFF
class adf1,adf2,cms1,cms2,AWS-account blueFill;

classDef orange stroke-width:2px,color:#FFFFFF,stroke:#FF9D1C
class adf1,adf2,cms1,cms2,AWS-account orange
```

### CloudWatch roles and permissions

To use CloudWatch Metric Streams in XCOR, you must configure a
CloudWatch Metric Stream in each AWS account and region. The account you use to set
up the CloudWatch Metric Stream must have either the `CloudWatchFullAccess` policy
and `iam:PassRole` permission or all of the following permissions:

* `iam:PassRole`
* `cloudwatch:PutMetricStream`
* `cloudwatch:DeleteMetricStream`
* `cloudwatch:GetMetricStream`
* `cloudwatch:ListMetricStreams`
* `cloudwatch:StartMetricStreams`
* `cloudwatch:StopMetricStreams`
* `iam:CreateRole`
* `iam:PutRolePolicy`

### XCOR authentication

Create or use an API token for an XCOR restricted service account with
write-only permission. For more information, see
[Create a restricted service account](https://docs-xcor.paloaltonetworks.com/administer/accounts-teams/service-accounts.md#create-a-restricted-service-account).

You must also provide your XCOR organization name. This is the `<ORG_NAME>`
segment in `https://<ORG_NAME>.chronosphere.io`.

### AWS resources and IAM roles

The setup creates the following resources and Identity and Access Management (IAM)
roles:

#### AWS resources

* Amazon S3 bucket: Stores eligible failed deliveries from Amazon Data Firehose.
* CloudWatch log group: Stores logs from the Firehose stream.
* Amazon Data Firehose stream: Sends metrics to XCOR through an HTTP
  endpoint and stores eligible failed deliveries in the S3 bucket.
* IAM role for Amazon Data Firehose: Grants the Firehose stream the following
  permissions to access the S3 bucket and CloudWatch Logs:
  * `s3:AbortMultipartUpload`
  * `s3:GetBucketLocation`
  * `s3:GetObject`
  * `s3:ListBucket`
  * `s3:ListBucketMultipartUploads`
  * `s3:PutObject`
  * `logs:PutLogEvents`

#### IAM roles

The setup also creates an IAM role with these permissions so CloudWatch Metric Streams
can publish data to the Firehose stream:

* `firehose:PutRecord`
* `firehose:PutRecordBatch`

### Apply the configuration

Choose either the Terraform setup or the AWS Management Console setup.

> [tab] Terraform
>
> 1. Set values for the following
>    [Terraform](https://docs-xcor.paloaltonetworks.com/tooling/infrastructure/terraform)
>    variables, and then apply the configuration.
>
>    ```terraform expandable Terraform example icon="square-terminal" theme={null}
>    # Variables
>
>    variable "chronosphere_org_name" {
>      type        = string
>      description = "The name of your Cortex XCOR organization, which is the subdomain name before .chronosphere.io."
>    }
>
>    variable "chronosphere_api_token" {
>      type        = string
>      sensitive   = true
>      description = "The API token for a Cortex XCOR Restricted Service Account with write-only permission."
>    }
>
>    variable "failed_data_bucket_name" {
>      type        = string
>      description = "The name of the S3 bucket to create to store data that couldn't be delivered to Cortex XCOR. If not specified, a random name will be generated."
>      default     = ""
>    }
>
>    variable "common_resource_attributes" {
>      type        = map(string)
>      description = "Key-value pairs to apply as OpenTelemetry Resource Attributes on all metrics in this stream."
>      default     = {}
>    }
>
>    # IAM Policy Documents
>
>    data "aws_iam_policy_document" "kinesis-firehose-stream-role-trust-policy" {
>      statement {
>        sid     = "AllowRoleAssumptionByKinesisFirehose"
>        effect  = "Allow"
>        actions = ["sts:AssumeRole"]
>
>        principals {
>          type        = "Service"
>          identifiers = ["firehose.amazonaws.com"]
>        }
>      }
>    }
>
>    data "aws_iam_policy_document" "kinesis-firehose-stream-role-s3-policy" {
>      statement {
>        sid     = "AllowFirehoseS3Access"
>        effect  = "Allow"
>        actions = [
>          "s3:AbortMultipartUpload",
>          "s3:GetBucketLocation",
>          "s3:GetObject",
>          "s3:ListBucket",
>          "s3:ListBucketMultipartUploads",
>          "s3:PutObject"
>        ]
>
>        resources = [
>          aws_s3_bucket.kinesis-firehose-stream-failed-data.arn,
>          "${aws_s3_bucket.kinesis-firehose-stream-failed-data.arn}/*"
>        ]
>      }
>
>      statement {
>        sid       = "AllowFirehoseCloudWatchLogs"
>        effect    = "Allow"
>        actions   = ["logs:PutLogEvents"]
>        resources = ["${aws_cloudwatch_log_group.kinesis-firehose-stream.arn}:*"]
>      }
>    }
>
>    data "aws_iam_policy_document" "cloudwatch-metric-stream-role-trust-policy" {
>      statement {
>        sid     = "AllowRoleAssumptionByCloudWatchMetricStream"
>        effect  = "Allow"
>        actions = ["sts:AssumeRole"]
>
>        principals {
>          type        = "Service"
>          identifiers = ["streams.metrics.cloudwatch.amazonaws.com"]
>        }
>      }
>    }
>
>    data "aws_iam_policy_document" "cloudwatch-metric-stream-role-firehose-policy" {
>      statement {
>        sid    = "AllowCloudWatchFirehoseAccess"
>        effect = "Allow"
>        actions = [
>          "firehose:PutRecord",
>          "firehose:PutRecordBatch"
>        ]
>
>        resources = [
>          aws_kinesis_firehose_delivery_stream.kinesis-firehose-stream.arn
>        ]
>      }
>    }
>
>    # S3
>
>    resource "random_id" "default_bucket_name_suffix" {
>      byte_length = 8
>    }
>
>    resource "aws_s3_bucket" "kinesis-firehose-stream-failed-data" {
>      bucket = var.failed_data_bucket_name != "" ? var.failed_data_bucket_name : "chronosphere-cw-stream-failed-data-${random_id.default_bucket_name_suffix.hex}"
>    }
>
>    resource "aws_s3_bucket_public_access_block" "kinesis-firehose-stream-failed-data" {
>      bucket                  = aws_s3_bucket.kinesis-firehose-stream-failed-data.id
>      block_public_acls       = true
>      block_public_policy     = true
>      ignore_public_acls      = true
>      restrict_public_buckets = true
>    }
>
>    resource "aws_s3_bucket_server_side_encryption_configuration" "kinesis-firehose-stream-failed-data" {
>      bucket = aws_s3_bucket.kinesis-firehose-stream-failed-data.id
>
>      rule {
>        apply_server_side_encryption_by_default {
>          sse_algorithm = "AES256"
>        }
>
>        bucket_key_enabled = false
>      }
>    }
>
>    resource "aws_s3_bucket_lifecycle_configuration" "kinesis-firehose-stream-failed-data" {
>      bucket = aws_s3_bucket.kinesis-firehose-stream-failed-data.id
>
>      rule {
>        id = "Cleanup"
>
>        expiration {
>          days = 90
>        }
>
>        status = "Enabled"
>      }
>    }
>
>    # CloudWatch Logs
>
>    resource "aws_cloudwatch_log_group" "kinesis-firehose-stream" {
>      name              = "/aws/kinesisfirehose/chronosphere-cloudwatch-firehose"
>      retention_in_days = 14
>    }
>
>    resource "aws_cloudwatch_log_stream" "kinesis-firehose-stream" {
>      name           = "HttpEndpointDelivery"
>      log_group_name = aws_cloudwatch_log_group.kinesis-firehose-stream.name
>    }
>
>    # Kinesis Firehose Delivery Stream
>
>    resource "aws_iam_role" "kinesis-firehose-stream-role" {
>      name               = "cloudwatch-firehose-stream-role"
>      assume_role_policy = data.aws_iam_policy_document.kinesis-firehose-stream-role-trust-policy.json
>
>      tags = {
>        Name = "cloudwatch-firehose-stream-role"
>      }
>    }
>
>    resource "aws_iam_role_policy" "kinesis-firehose-stream-role-s3-policy" {
>      name   = "KinesisFirehose-S3Access"
>      role   = aws_iam_role.kinesis-firehose-stream-role.id
>      policy = data.aws_iam_policy_document.kinesis-firehose-stream-role-s3-policy.json
>    }
>
>    resource "aws_kinesis_firehose_delivery_stream" "kinesis-firehose-stream" {
>      name        = "chronosphere-cloudwatch-firehose"
>      destination = "http_endpoint"
>
>      http_endpoint_configuration {
>        name               = "chronosphere-http-endpoint"
>        url                = "https://${var.chronosphere_org_name}.chronosphere.io/data/metrics/api/v1/cloudwatch/firehose"
>        access_key         = var.chronosphere_api_token
>        buffering_size     = 1  # MiB
>        buffering_interval = 60 # seconds
>        role_arn           = aws_iam_role.kinesis-firehose-stream-role.arn
>        s3_backup_mode     = "FailedDataOnly"
>        retry_duration     = 300 # seconds
>
>        s3_configuration {
>          role_arn           = aws_iam_role.kinesis-firehose-stream-role.arn
>          bucket_arn         = aws_s3_bucket.kinesis-firehose-stream-failed-data.arn
>          buffering_size     = 10  # MiB
>          buffering_interval = 300 # seconds
>          compression_format = "GZIP"
>        }
>
>        request_configuration {
>          content_encoding = "GZIP"
>
>          dynamic "common_attributes" {
>            for_each = var.common_resource_attributes
>            iterator = attribute
>            content {
>              name  = attribute.key
>              value = attribute.value
>            }
>          }
>        }
>
>        cloudwatch_logging_options {
>          enabled         = true
>          log_group_name  = aws_cloudwatch_log_group.kinesis-firehose-stream.name
>          log_stream_name = aws_cloudwatch_log_stream.kinesis-firehose-stream.name
>        }
>      }
>
>      server_side_encryption {
>        enabled = true
>      }
>    }
>
>    # CloudWatch Metric Stream
>
>    resource "aws_iam_role" "cloudwatch-metric-stream-role" {
>      name               = "cloudwatch-metric-stream-role"
>      assume_role_policy = data.aws_iam_policy_document.cloudwatch-metric-stream-role-trust-policy.json
>
>      tags = {
>        Name = "cloudwatch-metric-stream-role"
>      }
>    }
>
>    resource "aws_iam_role_policy" "cloudwatch-metric-stream-role-firehose-policy" {
>      name   = "MetricStreams-FirehosePutRecords"
>      role   = aws_iam_role.cloudwatch-metric-stream-role.id
>      policy = data.aws_iam_policy_document.cloudwatch-metric-stream-role-firehose-policy.json
>    }
>
>    resource "aws_cloudwatch_metric_stream" "cloudwatch-metric-stream" {
>      name          = "chronosphere-cloudwatch-metric-stream"
>      role_arn      = aws_iam_role.cloudwatch-metric-stream-role.arn
>      firehose_arn  = aws_kinesis_firehose_delivery_stream.kinesis-firehose-stream.arn
>      output_format = "opentelemetry1.0"
>    }
>    ```

> [tab] AWS Management Console
>
> To configure CloudWatch Metric Streams using the AWS Management Console:
>
> 1. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/).
> 2. In the navigation pane, go to **Metrics > Streams**.
> 3. Select **Create metric stream**.
> 4. Select **Custom setup with Firehose**.
> 5. To create an Amazon Data Firehose stream, select **Set up an Amazon Data Firehose stream**
>    under **Select your Amazon Data Firehose stream**.
> 6. In **Create Firehose Stream**, set the following options:
>    * **Set source:** `Direct PUT`
>    * **Destination:** `HTTP Endpoint`
>    * **Firehose stream name:** `chronosphere-cloudwatch-firehose`
>    * **Destination settings:**
>      * **HTTP endpoint name:** `Cortex XCOR CloudWatch Endpoint`
>      * **HTTP endpoint URL:** `https://<ORG_NAME>.chronosphere.io/data/metrics/api/v1/cloudwatch/firehose`.
>        Replace `<ORG_NAME>` with your XCOR organization name.
>      * **Authentication:** `Use access key`
>      * **Access key:** Enter your XCOR Restricted Service Account's
>        API token.
>      * **Content encoding:** `GZIP`
>      * **Retry duration:** `300 seconds`
>      * Expand **Buffering Hints**.
>        * **Buffer size:** `1 MiB`
>        * **Buffer interval:** `60 seconds`
>    * **Backup settings:**
>      * **Source record backup in Amazon S3:** `Failed data only`
>      * **S3 backup bucket:** Select a bucket, or create a new bucket. If you create
>        a new bucket, provide a unique name and leave all default settings.
>      * Expand **Buffer hints, compression and encryption**.
>        * **Buffer size:** `10 MiB`
>        * **Buffer interval:** `300 seconds`
>        * **Compression for data records:** `GZIP`
> 7. Select **Create Firehose stream**.
> 8. Return to **Create a metric stream**.
> 9. In **Select your Amazon Data Firehose stream**, select `chronosphere-cloudwatch-firehose`.
>    This is the Amazon Data Firehose stream you created previously.
> 10. Expand **Change output format** and verify that the output format is
>     `OpenTelemetry 1.0`.
> 11. Under **Metrics to be streamed**, select **All metrics** to stream all CloudWatch
>     metrics, or **Select metrics** to define which metrics to include or exclude in
>     the metrics stream.
> 12. Set the **Custom metric stream name** to `chronosphere-cloudwatch-metric-stream`.
> 13. Select **Create metric stream**.

## Validate CloudWatch Metric Stream ingestion

After setup, wait for data to arrive in XCOR. Validate ingestion with the
operational dashboards in XCOR and the AWS Management Console.

> [!NOTE]
>
> The included dashboards query the default CloudWatch metric names. For tenants
> configured for Datadog-compatible naming, validate the `snake_case` metric names in
> [Metrics Explorer](https://docs-xcor.paloaltonetworks.com/investigate/querying/metrics/explorer.md).

The integration also includes a Metrics Overview dashboard that shows CloudWatch Metric Streams
delivery volume and publish errors by account, region, and stream. The Metrics Overview dashboard
is available from the Amazon Web Services integration in the
[Integrations Hub](https://docs-xcor.paloaltonetworks.com/integrate/integrations-hub.md).

### Validate the CloudWatch Metrics Ingestion & Health dashboard

The CloudWatch Metrics Ingestion & Health dashboard displays the health of your
CloudWatch Metric Streams integration with XCOR.

The CloudWatch Metric Streams and Data Firehose panel groups rely on
CloudWatch metrics sent from those services to XCOR. To populate
these charts, include metrics from the `AWS/Firehose` and
`AWS/CloudWatch/MetricStreams` namespaces in your CloudWatch Metric Streams
configuration.

1. In XCOR, go to **Dashboards**.

2. In the search bar, enter **CloudWatch**, and then select the
   **CloudWatch Metrics Ingestion & Health** dashboard.

   The metrics ingestion panel group displays information about the CloudWatch
   metrics XCOR received.

3. Use the **Data Firehose records received by Amazon Resource Name** chart to
   confirm that the ingestion API received Data Firehose records.

4. Use the **CloudWatch metric updates received** chart to confirm the number of
   CloudWatch metric updates XCOR extracts from the Data Firehose
   records.

5. Confirm that the **Transformed metrics** chart shows no **Rejected Data Points**.

6. Use the **Unique time series by AWS metric namespace** chart to confirm that the
   stream received by XCOR includes metrics from the AWS namespaces you want.

### Validate the Metric Stream dashboard

In the AWS Management Console, use the **Metric Updates** chart for specific
metrics to validate that metrics are streaming.

1. In the AWS Management Console, go to
   **CloudWatch > Metric Streams**.
2. Select `chronosphere-cloudwatch-metric-stream` to view the status and operational
   statistics.
3. Verify that the **Status** is `Running`.
4. Use the **Metric Updates** chart to verify how many updates have been sent. If
   the stream is working, the chart should report a non-zero number of updates.
5. Verify whether any errors were reported in the **Errors** chart. The value should
   be 0.

### Validate the Amazon Data Firehose status

In the AWS Management Console, use the Firehose charts to validate that metrics are
streaming.

1. In the AWS Management Console, go to
   **Amazon Data Firehose > Firehose Streams**.
2. Select `chronosphere-cloudwatch-firehose` to view status and operational statistics
   for the Firehose stream.
3. The **Incoming bytes**, **Incoming put requests**, and **Incoming records** charts
   should all report non-zero values.
4. The **HTTP endpoint delivery success** chart should report 100%.
5. The **Records delivered to HTTP endpoint** chart should report a non-zero value.

### Query metrics about AWS CloudWatch Metric Streams

Query metrics about AWS CloudWatch Metric Streams in XCOR. These metrics don't
appear in XCOR Metrics Explorer
if a drop rule excludes all AWS metrics. Modify the rule to allow metrics from at
least one namespace, such as all metrics from the
`AWS/CloudWatch/MetricStreams` namespace. For examples, see
[Control metric volume](#control-metric-volume).

1. In XCOR, go to
   ** Explorers > Metrics Explorer**.
2. In the query box, enter `aws_cloudwatch_metricstreams` to view a list of AWS
   metrics received from the CloudWatch Metric Stream.
3. Select the metric you want to query to add it to the query prompt.

Use the selected metric in a query. For default metric naming, run this query
to report the rate of CloudWatch metric updates:

```text theme={null}
rate(aws_cloudwatch_metricstreams_MetricUpdate_sum[5m])
```

For Datadog-compatible metric naming, run:

```text theme={null}
rate(aws_cloudwatch_metricstreams_metric_update_sum[5m])
```

## Metric semantics

### Metric naming conventions

Ingested CloudWatch metric names in XCOR follow this pattern:

```text theme={null}
<namespace>_<MetricName>_<statistic>
```

* `<namespace>`: The namespace is lowercased, and XCOR replaces
  all forward slash (`/`) and period (`.`) characters in the CloudWatch namespace
  with underscores (`_`).

  All AWS service namespaces follow the naming convention `AWS/<ServiceName>`, where
  `<ServiceName>` is replaced with the service name. In XCOR,
  the ingested metrics therefore begin with `aws_<servicename>`. For a list of AWS
  services and their respective namespaces, see
  [AWS services that publish CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html).

  If you create custom metrics, the namespace you set for the metric becomes the
  metric name prefix in XCOR.
* `<MetricName>`: By default, XCOR preserves the CloudWatch metric name's
  case. Tenants configured for Datadog-compatible naming receive the metric name in
  `snake_case`.
* `<statistic>`: XCOR appends the CloudWatch statistic's name
  (`count`, `sum`, `maximum`, `minimum`, `average`). If you define additional statistics
  for a metric, XCOR appends the corresponding CloudWatch metric
  statistic name (`pXX`).

For examples, see [Example metric names](#example-metric-names).

### Metric labeling conventions

XCOR adds CloudWatch metric dimensions as labels to each time series,
following the pattern `dimension_<DimensionName>`. For examples, see
[Example metric names](#example-metric-names).

### Stream resource attributes

CloudWatch Metric Streams include OpenTelemetry Protocol (OTLP) resource attributes,
which XCOR merges into the time series. XCOR
replaces periods (`.`) with underscores (`_`) in attribute key names.

CloudWatch Metric Streams includes the following resource attributes in the data it
sends through Amazon Data Firehose:

* `aws_exporter_arn`: The Amazon Resource Name (ARN) of the CloudWatch Metric Stream,
  which uniquely identifies the metric writer instance.
* `cloud_account_id`: The AWS account associated with the CloudWatch Metric Stream,
  such as `123456789012`.
* `cloud_provider`: The value is always `aws`.
* `cloud_region`: The AWS region of the Amazon Data Firehose sending the stream,
  such as `us-east-2`.

For examples, see [Example metric names](#example-metric-names).

#### Add custom resource attributes using stream parameters

Define custom key-value pairs as parameters that Amazon Data Firehose includes in
each HTTP request. XCOR treats all additional parameters as resource
attributes and merges them into the time series.

> [!NOTE]
>
> Your custom parameters take precedence over the default CloudWatch resource
> attributes. Avoid parameter names that conflict with default CloudWatch attribute
> names.

Configure additional parameters through Terraform or the AWS Management Console.

> [tab] Terraform
>
> To configure parameters using Terraform:
>
> 1. Edit the Terraform module.
> 2. In the `aws_kinesis_firehose_delivery_stream` resource definition, modify the
>    `request_configuration` block to define additional `common_attributes`. The
>    AWS Management Console names this setting **Parameters**, while the API name is
>    `common_attributes`.
>
>    The following example configures the `content_encoding` parameter as `GZIP` and
>    defines two `common_attributes`, `testname` and `testname2`:
>
>    ```terraform Terraform example icon="square-terminal" theme={null}
>    request_configuration {
>      content_encoding = "GZIP"
>
>      common_attributes {
>        name  = "testname"
>        value = "testvalue"
>      }
>
>      common_attributes {
>        name  = "testname2"
>        value = "testvalue2"
>      }
>    }
>    ```

> [tab] AWS Management Console
>
> To configure parameters using the AWS Management Console:
>
> 1. Navigate to the Amazon Data Firehose service.
> 2. Select the stream sending CloudWatch metrics
>    (`chronosphere-cloudwatch-firehose`).
> 3. Select the **Configuration** tab in the Firehose stream details view.
> 4. In the **Destination settings** panel, select **Add parameters**.
> 5. In the **Edit destination settings** view, select **Add parameter**.
> 6. Enter the key and value. To add more parameters, select **Add parameter**.
> 7. Select **Save changes**.

### Example metric names

Given a CloudWatch metric with the following attributes:

* The namespace `AWS/EBS` (AWS Service)
* The metric name `VolumeReadBytes`
* The dimension `VolumeId`
* The custom Firehose destination parameter `environment`
* The CloudWatch metric resource attributes `aws_exporter_arn`, `cloud_account_id`,
  `cloud_provider`, and `cloud_region`

The following examples focus on the dimension and resource attributes. Every series
also carries `Namespace`, `MetricName`, and `instance` (the metric stream's ARN)
labels.

XCOR creates metrics with these names and labels:

```text theme={null}
aws_ebs_VolumeReadBytes_count{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
aws_ebs_VolumeReadBytes_sum{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
aws_ebs_VolumeReadBytes_maximum{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
aws_ebs_VolumeReadBytes_minimum{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
aws_ebs_VolumeReadBytes_average{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
```

Given a CloudWatch metric with the following attributes:

* The namespace `Buildkite` (custom metrics)
* The metric name `RunningJobsCount`
* The custom Firehose destination parameter `environment`
* No dimensions
* CloudWatch metric resource attributes `aws_exporter_arn`, `cloud_account_id`,
  `cloud_provider`, and `cloud_region`

XCOR creates metrics with these names and labels:

```text theme={null}
buildkite_RunningJobsCount_count{aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
buildkite_RunningJobsCount_sum{aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
buildkite_RunningJobsCount_maximum{aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
buildkite_RunningJobsCount_minimum{aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
buildkite_RunningJobsCount_average{aws_exporter_arn="abc123", cloud_account_id="123456789012", cloud_provider="aws", cloud_region="us-east-2", environment="staging"}
```

## S3 daily storage metrics

Amazon S3 publishes its daily storage metrics, `BucketSizeBytes` and
`NumberOfObjects`, to CloudWatch once a day, and they don't arrive in
XCOR through CloudWatch Metric Streams. XCOR can read them from
CloudWatch in your AWS accounts instead, with the same AWS read role it uses to
read your resource tags.

> [!NOTE]
>
> XCOR Support turns on these reads for your tenant. To request them, contact
> [XCOR Support](https://docs-xcor.paloaltonetworks.com/support.md).

XCOR excludes both metrics from the aggregation recommendations for
every tenant, because their values arrive after the window aggregation uses.

### Prerequisites for S3 daily storage metrics

XCOR reads the AWS accounts and regions where it reads your AWS resource
tags directly from AWS. Each account needs the XCOR AWS read role at
template version 1.1.0 or later, which grants the read-only
`cloudwatch:ListMetrics` and `cloudwatch:GetMetricData` actions these reads use.

* AWS bills `GetMetricData` requests to your account per metric requested.
  XCOR requests each bucket's metrics once or twice a day.
* S3 publishes a bucket's storage metrics only in the bucket's own region.
  XCOR reads no buckets in a region it doesn't read your resource tags in.

### S3 daily storage metric names

| Metric | Value | Unit |
| - | - | - |
| `aws_s3_bucket_size_bytes_average` | The data a bucket stores in one storage class, from `BucketSizeBytes` with the `Average` statistic. | Bytes |
| `aws_s3_number_of_objects_average` | The objects a bucket stores in every storage class, from `NumberOfObjects` with the `Average` statistic. | Count |

Each time series has these labels:

* `bucketname`: The bucket's name.
* `storagetype`: The storage class, such as `StandardStorage` or
  `GlacierStorage`. For `aws_s3_number_of_objects_average`, it's always
  `AllStorageTypes`.
* `Namespace` (always `AWS/S3`), `cloud_provider` (always `aws`),
  `cloud_account_id` and `cloud_region`.
* `aws_account_alias`: The alias XCOR Support set for the account, or
  else its AWS account alias. It's absent if the account has neither, or if the
  account's read role isn't allowed `iam:ListAccountAliases`, which read role
  template 1.2.0 grants.
* The bucket's resource tags, where XCOR reads them.

These time series don't have `dimension_*` labels or the stream resource
attributes that streamed metrics have.

### S3 daily storage timestamps

XCOR writes each UTC day's value with a timestamp one hour after the day
ends. For example, the value for November 16 has the timestamp November 17,
01:00 UTC. CloudWatch timestamps the same value at the start of its day, so
the CloudWatch console shows it a day earlier than XCOR does.

XCOR reads each day once or twice between about 01:05 and 03:00 UTC the
next day. After XCOR Support turns on these reads, the first value
arrives between about 01:05 and 02:05 UTC the next day, for the day that was in
progress when they were turned on. If they're turned on before about 03:00 UTC,
XCOR might also write the value for the day before. It doesn't write
values for earlier days.

If AWS publishes a day's value later than about 01:55 UTC the next day, the
day might be missing. If AWS changes a day's value after that, XCOR keeps
the value it read.

### Query S3 daily storage metrics

Each value has one data point a day, so queries need a lookback of more than a
day:

* To show each day's value at the end of its day in a time series panel, and to
  line it up with other metrics, add `offset -1h` to the selector:

  ```text theme={null}
  sum by (bucketname) (aws_s3_bucket_size_bytes_average{storagetype="StandardStorage"} offset -1h)
  ```

* To show the latest value in a stat panel, a table, or a monitor, use
  `last_over_time` over 27 hours, with no offset. A day's value can arrive up
  to about two hours after its timestamp, so a window of one day returns no
  data each day from 01:00 UTC until that day's value arrives. An instant query
  without `last_over_time` returns no data for most of each day.

  ```text theme={null}
  sum by (bucketname) (last_over_time(aws_s3_bucket_size_bytes_average{storagetype="StandardStorage"}[27h]))
  ```

### Validate S3 daily storage ingestion

After 03:10 UTC on the day after XCOR Support turns on these reads, run
this query. It returns the number of buckets with a value in the last 27 hours:

```text theme={null}
count(last_over_time(aws_s3_number_of_objects_average[27h]))
```

## DynamoDB table size and item count metrics

Amazon DynamoDB reports each table's size and item count through its
`DescribeTable` API. These values aren't CloudWatch metrics, so CloudWatch
Metric Streams can't deliver them. XCOR can read them from DynamoDB in your AWS
accounts instead, with the same AWS read role it uses to read your resource
tags.

> [!NOTE]
>
> XCOR Support turns on these reads for your tenant. To request them, contact
> [XCOR Support](https://docs-xcor.paloaltonetworks.com/support.md).

### Prerequisites for DynamoDB table metrics

XCOR reads the AWS accounts and regions where it reads your AWS resource
tags directly from AWS. Each account needs the XCOR AWS read role at
template version 1.1.0 or later, which grants the read-only
`dynamodb:ListTables` and `dynamodb:DescribeTable` actions these reads use.

* In each account and region, XCOR lists your tables with `ListTables` and
  calls `DescribeTable` for each table it lists, up to 10,000 tables. If a
  call fails, for example because AWS still throttles it after retries, the
  tables after it get no value from that read.
* XCOR writes no values for a table whose resource policy denies the read role
  `dynamodb:DescribeTable`.

### DynamoDB table metric names

| Metric | Value | Unit |
| - | - | - |
| `aws_dynamodb_table_size` | The table's size, from `TableSizeBytes`. | Bytes |
| `aws_dynamodb_item_count` | The number of items in the table, from `ItemCount`. | Count |

Both metrics are gauges. Each time series has these labels:

* `dimension_TableName`: The table's name. Streamed `AWS/DynamoDB` metrics that
  have a `TableName` dimension carry the same label.
* `Namespace` (always `AWS/DynamoDB`), `cloud_provider` (always `aws`),
  `cloud_account_id` and `cloud_region`.
* `aws_account_alias`: The alias configured for the account in XCOR, or else
  its AWS account alias. It's absent if the account has neither, or if the
  account's read role isn't allowed `iam:ListAccountAliases`, which read role
  template 1.2.0 grants. It can also be absent for a while when XCOR can't read
  the AWS account alias, and appears once a read succeeds. The label is part of
  each time series' identity, so an account with no alias configured gets new
  time series when its AWS account alias appears, changes, or goes away.
* The table's resource tags, where XCOR reads them.

These time series don't have the stream resource attributes that streamed
metrics have.

XCOR writes no values for a table that's being archived or is archived, or
for a table whose AWS KMS key DynamoDB can't access
(`INACCESSIBLE_ENCRYPTION_CREDENTIALS`). For an inaccessible key, values
resume once DynamoDB can access the key again.

### DynamoDB table metric timestamps

XCOR reads your tables every minute by default, and timestamps each value at
the time it reads it. XCOR Support can set a longer interval, up to four
minutes.

DynamoDB updates `TableSizeBytes` and `ItemCount` approximately every six
hours, so each time series holds the same value for up to six hours and then
changes in a single step. Recent writes to a table don't appear in these
metrics until DynamoDB updates the values, and a new table can read 0 bytes
and 0 items until DynamoDB first updates them.

### Query DynamoDB table metrics

Each table's values change only when DynamoDB updates them, so compare values
over periods longer than six hours:

* To show the ten largest tables:

  ```text theme={null}
  topk(10, aws_dynamodb_table_size)
  ```

* To show how much each table grew over the last day:

  ```text theme={null}
  aws_dynamodb_table_size - aws_dynamodb_table_size offset 1d
  ```

* To show the total items in each account:

  ```text theme={null}
  sum by (cloud_account_id) (aws_dynamodb_item_count)
  ```

### Validate DynamoDB table metric ingestion

After XCOR Support turns on these reads, run this query. It returns the
number of tables with a value in the last 10 minutes:

```text theme={null}
count(last_over_time(aws_dynamodb_table_size[10m]))
```

## Lambda function configuration metrics

Each AWS Lambda function's configured timeout and memory size are settings, not
CloudWatch metrics, so CloudWatch Metric Streams can't send them. XCOR can
read them from your AWS accounts with the Lambda `ListFunctions` API, using the
same AWS read role it uses to read your resource tags, and write them as two
metrics, `aws_lambda_timeout` and `aws_lambda_memorysize`.

Dashboards and monitors converted from Datadog that query `aws.lambda.timeout`
and `aws.lambda.memorysize` read these two metrics with no change.

> [!NOTE]
>
> XCOR Support turns on these reads for your tenant. To request them, contact
> [XCOR Support](https://docs-xcor.paloaltonetworks.com/support.md).

XCOR reads each function's unpublished (`$LATEST`) configuration, for up to
10,000 functions in each account and region. It doesn't read published versions
or aliases. From each function, XCOR copies its name,
timeout, memory size and architecture. `ListFunctions` also returns each
function's environment variables, which XCOR doesn't read, log, store or
write.

### Prerequisites for Lambda function configuration metrics

XCOR reads the AWS accounts and regions where it reads your AWS resource
tags directly from AWS. There's nothing else to configure for Lambda. Each
account needs the XCOR AWS read role at template version 1.1.0 or later, which
grants the read-only `lambda:ListFunctions` action these reads use. An account
whose role is older denies the action, and XCOR writes no Lambda metrics for
it.

### Lambda function configuration metric names

| Metric | Value | Unit |
| - | - | - |
| `aws_lambda_timeout` | The function's configured timeout, from `Timeout` in its configuration. | Seconds |
| `aws_lambda_memorysize` | The function's configured memory, from `MemorySize` in its configuration. | Mebibytes |

Both metrics are gauges. XCOR writes the values as Lambda returns them, with no
unit conversion: Lambda's "MB" is a mebibyte. A function whose configuration
omits one of the values has no sample of that metric until a later read returns
it. The names are the same for tenants configured for Datadog-compatible naming.

Each time series has these labels:

* `dimension_FunctionName`: The function's name, such as `orders-api`.
* `architecture`: The function's instruction set architecture, `x86_64` or
  `arm64`. It's absent if Lambda returns other than one architecture for the
  function.
* `Namespace` (always `AWS/Lambda`), `cloud_provider` (always `aws`),
  `cloud_account_id` and `cloud_region`.
* `aws_account_alias`: The alias XCOR Support set for the account, or
  else its AWS account alias. It's absent if the account has neither, or if the
  account's read role isn't allowed `iam:ListAccountAliases`, which read role
  template 1.2.0 grants. The label is part of each time series' identity, so
  an account with no alias set by XCOR Support gets new time series when
  its AWS account alias appears, changes or goes away.
* `xcor.integration.name`: Always `aws`. The name contains dots, so quote it in
  a selector: `{"xcor.integration.name"="aws"}`.
* The function's resource tags, where XCOR reads them.

These time series don't have the `dimension_Resource` or
`dimension_ExecutedVersion` labels that some streamed Lambda metrics have,
because XCOR reads only each function's unpublished configuration. They don't
have the stream resource attributes that streamed metrics have.

### Lambda function configuration timestamps

XCOR reads every function in each account and region once a minute by default,
and writes one sample of each metric per function. Each sample's timestamp is
the time XCOR finished reading the function's account and region. A change to a
function's configuration appears in XCOR at the next read.

### Query Lambda function configuration metrics

Each function has a sample about once a minute, so instant queries and range
queries work as they do for streamed metrics:

```text theme={null}
max by (dimension_FunctionName) (aws_lambda_memorysize{cloud_account_id="123456789012"})
```

Functions in different accounts or regions can share a name. To keep them
apart, add `cloud_account_id` and `cloud_region` to the labels you aggregate
by.

The timeout is in seconds, and the streamed Lambda `Duration` metric is in
milliseconds. To compare one function's longest run with its timeout, multiply
the timeout by 1,000:

```text theme={null}
max(aws_lambda_Duration_maximum{dimension_FunctionName="orders-api"}) / (max(aws_lambda_timeout{dimension_FunctionName="orders-api"}) * 1000)
```

Tenants configured for Datadog-compatible naming receive the streamed metric as
`aws_lambda_duration_maximum`.

### Validate Lambda function configuration ingestion

A few minutes after XCOR Support turns on these reads, run this query. It
returns the number of functions with a timeout value in the last 10 minutes:

```text theme={null}
count(count by (cloud_account_id, cloud_region, dimension_FunctionName) (last_over_time(aws_lambda_timeout[10m])))
```

### Troubleshoot Lambda function configuration metrics

* **No Lambda metrics for one account.** Check the account's read role version.
  A role older than 1.1.0 denies `lambda:ListFunctions`.
* **Gaps during busy periods.** `ListFunctions` shares Lambda's control-plane
  request quota of 15 requests a second in each region with your other Lambda
  management calls. When Lambda throttles a read, XCOR writes the functions it
  read before the throttle and reads all of them again at the next read.
* **A function has no `architecture` label.** Lambda returned other than one
  architecture for it. Its two metrics are still written.
* **Some functions in a large region have no metrics.** XCOR reads at most
  10,000 functions in each account and region. In a region with more, the
  functions after the first 10,000 that Lambda lists get no metrics.

## Amazon ECS pending tasks

XCOR can read two counts of Amazon Elastic Container Service (Amazon ECS)
tasks in the `PENDING` state from the ECS API in your AWS accounts, with the
same AWS read role it uses to read your resource tags. Neither count is in the
`AWS/ECS` CloudWatch namespace, so CloudWatch Metric Streams can't deliver
them. CloudWatch Container Insights, where you turn it on, publishes its own
per-service `PendingTaskCount`, which is a separate metric.

> [!NOTE]
>
> XCOR Support turns on these reads for your tenant. To request them, contact
> [XCOR Support](https://docs-xcor.paloaltonetworks.com/support.md).

For each account and region, XCOR lists your clusters with `ListClusters`. In
each cluster, it lists the container instances with `ListContainerInstances`
and describes them with `DescribeContainerInstances`, and lists the services
with `ListServices` and describes them with `DescribeServices`.

### Prerequisites for Amazon ECS pending tasks

XCOR reads the AWS accounts and regions where it reads your AWS resource
tags directly from AWS. Each account needs the XCOR AWS read role at
template version 1.1.0 or later, which grants the read-only actions these reads
use:

* `ecs:ListClusters`
* `ecs:ListContainerInstances`
* `ecs:DescribeContainerInstances`
* `ecs:ListServices`
* `ecs:DescribeServices`

Amazon ECS limits the rate of API calls in each account and region, and the
limit is shared with your own deployments and tools. Each XCOR read of an
account and region makes at most five ECS calls a second, a quarter or less of
each limit's refill rate.

### Amazon ECS pending tasks metric names

| Metric | Value | Unit |
| - | - | - |
| `aws_ecs_pending_tasks_count` | The tasks in the `PENDING` state on one container instance, from `pendingTasksCount` in [`DescribeContainerInstances`](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeContainerInstances.html). | Tasks |
| `aws_ecs_service_pending_sum` | The tasks in the `PENDING` state in one service, from `pendingCount` in [`DescribeServices`](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeServices.html). | Tasks |

Both metrics are gauges. The `_sum` in `aws_ecs_service_pending_sum` is part
of the name, which matches dashboards converted from Datadog's
`aws.ecs.service.pending`. The value is the service's current count, not a sum
over time.

Each time series has these labels:

* `dimension_ClusterName`: The cluster's name.
* `dimension_InstanceId`: On `aws_ecs_pending_tasks_count` only. The container
  instance's Amazon EC2 instance ID, such as `i-0123456789abcdef0`. For an
  external instance registered with ECS Anywhere, the managed instance ID, such
  as `mi-0123456789abcdef0`.
* `dimension_ServiceName`: On `aws_ecs_service_pending_sum` only. The
  service's name.
* `Namespace` (always `AWS/ECS`), `cloud_provider` (always `aws`),
  `cloud_account_id` and `cloud_region`.
* `xcor.integration.name`: Always `aws`.
* `aws_account_alias`: The alias XCOR Support set for the account, or
  else its AWS account alias. It's absent if the account has neither, or if the
  account's read role isn't allowed `iam:ListAccountAliases`, which read role
  template 1.2.0 grants. A permissions boundary or service control policy on the
  role must also allow it. It can also be absent for a while when XCOR can't read
  the AWS account alias, for example while IAM throttles the request. The label
  is part of each time series' identity, so an account with no alias set by
  XCOR Support gets new time series when its AWS account alias appears,
  changes, or goes away.
* Resource tags, where XCOR reads them: the service's tags on
  `aws_ecs_service_pending_sum`, or the cluster's tags for a service that has
  none, and the cluster's tags on `aws_ecs_pending_tasks_count`. A tag never
  replaces a label the time series already has.

These time series don't have the `MetricName`, `instance` or
`aws_exporter_arn` labels that streamed metrics have, or the custom resource
attributes you add with stream parameters. The example drop rule in
[Control metric volume](#control-metric-volume) filters on `aws_exporter_arn`,
so it doesn't drop them.

XCOR writes no value for a service that's being deleted (`DRAINING`) or is
deleted (`INACTIVE`), for a container instance that's deregistered
(`INACTIVE`), or for a container instance whose description has no
`ec2InstanceId`. When ECS can't describe a container instance or a service and
reports it in the response's `failures` list, XCOR writes no value for it on
that read. When a container instance is deregistered or a service is deleted,
its time series stops.

### Amazon ECS pending tasks timestamps

XCOR reads every cluster every four minutes by default. Each value is the
count when XCOR read it, with the timestamp of the time XCOR finished reading
that account and region, which can be minutes later in a large account.

### Query Amazon ECS pending tasks metrics

Each time series has one data point per read, so read a range longer than the
read interval, such as 10 minutes, with `last_over_time`. For example, this
query returns the pending tasks in each cluster's services:

```text theme={null}
sum by (cloud_account_id, cloud_region, dimension_ClusterName) (last_over_time(aws_ecs_service_pending_sum[10m]))
```

A container instance runs tasks from any service, so
`aws_ecs_pending_tasks_count` has no `dimension_ServiceName` label. A selector
such as `dimension_ServiceName=~".*"` still matches it, but one that names a
service doesn't.

The `xcor.integration.name` label and your AWS resource tag labels
(`aws.tag.<key>`) contain dots, so quote them in a selector:

```text theme={null}
aws_ecs_service_pending_sum{"xcor.integration.name"="aws"}
```

### Validate Amazon ECS pending tasks ingestion

After XCOR Support turns on these reads, allow up to about 20 minutes for
the first values to arrive.

1. In XCOR, go to
   ** Explorers > Metrics Explorer**.
2. Run this query to count the services read in each account and region:

   ```text theme={null}
   count by (cloud_account_id, cloud_region) (max by (cloud_account_id, cloud_region, dimension_ClusterName, dimension_ServiceName) (last_over_time(aws_ecs_service_pending_sum[10m])))
   ```

   For container instances, run:

   ```text theme={null}
   count by (cloud_account_id, cloud_region) (max by (cloud_account_id, cloud_region, dimension_InstanceId) (last_over_time(aws_ecs_pending_tasks_count[10m])))
   ```

   The queries read 10 minutes so that a single missed read doesn't hide a
   resource. Each account and region with ECS services, or with container
   instances, should appear. Each resource's newest value should be no older
   than one read interval, four minutes by default, plus the time XCOR takes to
   read all your accounts and regions, which can be several minutes when you
   have many.

If an account or region is missing, check that its read role allows the five
ECS actions in
[Prerequisites for Amazon ECS pending tasks](#prerequisites-for-amazon-ecs-pending-tasks).

## Amazon RDS allocated storage

XCOR can read the storage allocated to each Amazon Relational Database Service
(Amazon RDS) DB instance from the RDS API in your AWS accounts, with the same
AWS read role it uses to read your resource tags, and write it as the
`aws_rds_total_storage_space` metric. The allocated storage is a setting in
the instance's description, not a CloudWatch metric, so CloudWatch Metric
Streams can't deliver it.

> [!NOTE]
>
> XCOR Support turns on these reads for your tenant. To request them, contact
> [XCOR Support](https://docs-xcor.paloaltonetworks.com/support.md).

For each account and region, XCOR lists your DB instances with
[`DescribeDBInstances`](https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_DescribeDBInstances.html),
reading every page of results, and reads each instance's `AllocatedStorage`
setting. If a request for a later page fails, the instances on the pages before
it still get values from that read.

### Prerequisites for Amazon RDS allocated storage

XCOR reads the AWS accounts and regions where it reads your AWS resource
tags directly from AWS. Each account needs the XCOR AWS read role at
template version 1.1.0 or later, which grants the read-only
`rds:DescribeDBInstances` action these reads use.

* If your accounts run template 1.0.0, update the stack or StackSet to 1.1.0
  or later. The update changes the role's policy in place, and the role's
  Amazon Resource Name (ARN) doesn't change.
* If a permissions boundary or service control policy applies to the role, it
  must also allow `rds:DescribeDBInstances`. AWS doesn't report an error at
  deploy time when a boundary blocks an action.

XCOR reads each region of an account separately. Where the role isn't allowed
the action, AWS refuses the read, and XCOR writes no RDS allocated storage for
that account and region. A service control policy with a region condition can
refuse the read in some of an account's regions and allow it in others.
Accounts and regions whose roles allow it aren't affected.

### Amazon RDS allocated storage metric names

| Metric | Value | Unit |
| - | - | - |
| `aws_rds_total_storage_space` | The storage allocated to the DB instance. `DescribeDBInstances` returns `AllocatedStorage` in gibibytes (GiB), and XCOR multiplies it by 1,073,741,824, the number of bytes in a GiB. | Bytes |

The metric is a gauge, with one time series for each DB instance. It has no
statistic suffix, and the name is the same for tenants configured for
Datadog-compatible naming.

Each time series has these labels:

* `dimension_DBInstanceIdentifier`: The DB instance identifier. Streamed
  per-instance `AWS/RDS` metrics carry the same label.
* `dimension_EngineName`: The database engine, such as `postgres` or `mysql`.
* `dimension_DatabaseClass`: The DB instance class, such as `db.r6g.large`.
* `dimension_DBClusterIdentifier`: The Multi-AZ DB cluster the instance
  belongs to. It's absent if the instance isn't a member of one.
* `name`: The DB instance identifier, the same value as
  `dimension_DBInstanceIdentifier`.
* `dbname`: The instance's initial database name. It's absent if none was set.
* `Namespace` (always `AWS/RDS`), `cloud_provider` (always `aws`),
  `cloud_account_id` and `cloud_region`.
* `xcor.integration.name`: Always `aws`.
* `aws_account_alias`: The alias XCOR Support set for the account, or
  else its AWS account alias. It's absent if the account has neither, or if the
  account's read role isn't allowed `iam:ListAccountAliases`, which read role
  template 1.2.0 grants. A permissions boundary or service control policy on the
  role must also allow it. It can also be absent for a while when XCOR can't read
  the AWS account alias, for example while IAM throttles the request. The label
  is part of each time series' identity, so an account with no alias set by
  XCOR Support gets new time series when its AWS account alias appears,
  changes, or goes away.
* The instance's resource tags, where XCOR reads them. A tag never replaces a
  label the time series already has, such as a tag whose label name is `name`.
  If adding an instance's tags would exceed the label limits for a time series,
  XCOR writes that time series without any of its tags. If XCOR can't read your
  resource tags in time for a read, that read's values arrive without tags.
  Where XCOR Support turns on RDS attributes for your resource tags,
  these also include the instance's `engine`, its `dbclusteridentifier`, and
  its Availability Zone as `dimension_AvailabilityZone`.

Every label is part of a time series' identity, so a change to an instance's
class or tags, or a Multi-AZ failover that moves it to another Availability
Zone, starts a new time series. The old one stays in queries that read a range
including its last value. To count each instance once, aggregate by
`dimension_DBInstanceIdentifier` with `max`, as the queries in this section do.

The `dimension_` labels have the names streamed RDS metrics use, but they
aren't combined the same way. CloudWatch publishes an instance's streamed RDS
metrics with `dimension_DBInstanceIdentifier` only, and publishes engine-wide
and class-wide aggregates as separate time series. Each
`aws_rds_total_storage_space` time series carries the instance's engine, class
and cluster together.

These time series don't have the `MetricName`, `instance` or
`aws_exporter_arn` labels that streamed metrics have, or the custom resource
attributes you add with stream parameters. The example drop rule in
[Control metric volume](#control-metric-volume) filters on `aws_exporter_arn`,
so it doesn't drop them.

XCOR writes no value for an Amazon Aurora DB instance. An Aurora instance
stores its data in its cluster's shared storage volume, and AWS documents
`AllocatedStorage` as not applying to Aurora instances in
[`CreateDBInstance`](https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_CreateDBInstance.html),
so its value isn't the instance's storage. `aws_rds_total_storage_space` is
never an Aurora cluster's volume size. To monitor Aurora storage, use the
cluster's `VolumeBytesUsed` metric, which CloudWatch Metric Streams deliver
when your stream includes the `AWS/RDS` namespace. For details, see
[Amazon CloudWatch metrics for Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraMonitoring.Metrics.html).

XCOR also writes no value for Amazon Neptune and Amazon DocumentDB instances,
which `DescribeDBInstances` also returns, or for an instance that
`DescribeDBInstances` returns without an `AllocatedStorage` value. When an
instance is deleted, its time series stops.

### Amazon RDS allocated storage timestamps

XCOR reads every DB instance every two minutes by default. XCOR Support
can set an interval from one to four minutes. Each value is the instance's
allocated storage when XCOR read it, with the timestamp of the time XCOR
finished reading that account and region. A change to an instance's allocated
storage appears in XCOR at the first read after `DescribeDBInstances` returns
the new value.

### Query Amazon RDS allocated storage

Each time series has one data point per read, and its newest value can be as
old as one read interval plus the time XCOR takes to read all your accounts and
regions. Read a range longer than that, such as 10 minutes, with
`last_over_time`, rather than an instant query, which can return nothing for
some instances when XCOR takes several minutes to read your accounts.

DB instances in different accounts or regions can share an identifier, so keep
`cloud_account_id` and `cloud_region` in the labels you aggregate by. For
example, this query returns the storage allocated to all your DB instances in
each account and region:

```text theme={null}
sum by (cloud_account_id, cloud_region) (max by (cloud_account_id, cloud_region, dimension_DBInstanceIdentifier) (last_over_time(aws_rds_total_storage_space[10m])))
```

A query that combines a streamed metric with `aws_rds_total_storage_space`
must aggregate each side to the labels both have first, because the streamed
metric has labels this one doesn't. A division of the raw time series matches
nothing. For example, this query returns the percentage of each DB instance's
allocated storage that's free, from the streamed `FreeStorageSpace` metric,
which is also in bytes, with default metric naming:

```text theme={null}
100 * max by (cloud_account_id, cloud_region, dimension_DBInstanceIdentifier) (aws_rds_FreeStorageSpace_average)
  / max by (cloud_account_id, cloud_region, dimension_DBInstanceIdentifier) (last_over_time(aws_rds_total_storage_space[10m]))
```

For Datadog-compatible metric naming, run:

```text theme={null}
100 * max by (cloud_account_id, cloud_region, dimension_DBInstanceIdentifier) (aws_rds_free_storage_space_average)
  / max by (cloud_account_id, cloud_region, dimension_DBInstanceIdentifier) (last_over_time(aws_rds_total_storage_space[10m]))
```

The `xcor.integration.name` label and your AWS resource tag labels
(`aws.tag.<key>`) contain dots, so quote them in a selector:

```text theme={null}
aws_rds_total_storage_space{"xcor.integration.name"="aws"}
```

### Validate Amazon RDS allocated storage ingestion

After XCOR Support turns on these reads, allow up to about 20 minutes for
the first values to arrive.

1. In XCOR, go to
   ** Explorers > Metrics Explorer**.
2. Run this query to count the DB instances read in each account and region:

   ```text theme={null}
   count by (cloud_account_id, cloud_region) (max by (cloud_account_id, cloud_region, dimension_DBInstanceIdentifier) (last_over_time(aws_rds_total_storage_space[10m])))
   ```

   The query reads 10 minutes so that a single missed read doesn't hide an
   instance. Each account and region with RDS DB instances other than Aurora,
   Neptune and DocumentDB instances should appear. Each instance's newest value
   should be no older than one read interval, two minutes by default, plus the
   time XCOR takes to read all your accounts and regions, which can be several
   minutes when you have many.

If an account or region is missing, check that its read role allows
`rds:DescribeDBInstances`, as
[Prerequisites for Amazon RDS allocated storage](#prerequisites-for-amazon-rds-allocated-storage)
describes.

## Amazon EC2 host status and instance age

Each running Amazon EC2 instance's system status and launch time are fields of
the instance's description, not CloudWatch metrics, so CloudWatch Metric Streams
can't send them. XCOR can read them from your AWS accounts with the EC2
`DescribeInstanceStatus` and `DescribeInstances` APIs, using the same AWS read
role it uses to read your resource tags, and write them as two metrics,
`aws_ec2_host_ok` and `aws_ec2_instance_age`.

Dashboards and monitors converted from Datadog that query
`aws.ec2.instance_age` read `aws_ec2_instance_age`. Converted queries for
`aws.ec2.host_ok` read the streamed `aws_ec2_status_check_failed_average`
metric instead, so they don't read `aws_ec2_host_ok`. That's the streamed
metric's name for tenants configured for Datadog-compatible naming. Other
tenants receive it as `aws_ec2_StatusCheckFailed_average`.

> [!NOTE]
>
> XCOR Support turns on these reads for your tenant. To request them, contact
> [XCOR Support](https://docs-xcor.paloaltonetworks.com/support.md).

In each account and region, XCOR lists every running instance with
`DescribeInstances` and reads its system status with `DescribeInstanceStatus`.
Instances that are pending, stopping, stopped, shutting down or terminated
aren't listed, so their metrics stop until the instance runs again. Instances
that AWS services manage for you are included.

### Prerequisites for Amazon EC2 host status and instance age

XCOR reads the AWS accounts and regions where it reads your AWS resource
tags directly from AWS. There's nothing else to configure for EC2. Each account
needs the XCOR AWS read role at template version 1.1.0 or later, which grants
the read-only actions these reads use:

* `ec2:DescribeInstances`
* `ec2:DescribeInstanceStatus`

An account whose role is older denies both actions, and XCOR writes no EC2 host
metrics for it.

### Amazon EC2 host status and instance age metric names

| Metric | Value | Unit |
| - | - | - |
| `aws_ec2_host_ok` | 0 when the instance's system status is `impaired`. 1 when it's `ok`, `insufficient-data`, `initializing` or `not-applicable`, or when the instance has no system status. From `SystemStatus` in [`DescribeInstanceStatus`](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceStatus.html). | None (0 or 1) |
| `aws_ec2_instance_age` | Whole seconds since the instance was last launched, from `LaunchTime` in [`DescribeInstances`](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html). 0 if clock skew puts the launch after the read. | Seconds |

Both metrics are gauges. `aws_ec2_host_ok` follows only the system status: a
failing instance status check doesn't change it, and a new instance reads 1
while its status is `initializing`. The age is measured from the instance's last
launch, so a stop and start resets it. An instance whose description has no
launch time has no `aws_ec2_instance_age` sample from that read.

Each time series has these labels:

* `dimension_InstanceId`: The instance ID, such as `i-0123456789abcdef0`.
* `dimension_InstanceType`: The instance type, such as `m5.large`. It's absent
  from an `aws_ec2_host_ok` sample for an instance that started during the read.
* `dimension_AvailabilityZone`: The instance's Availability Zone, such as
  `us-east-1a`.
* `Namespace` (always `AWS/EC2`), `cloud_provider` (always `aws`),
  `cloud_account_id` and `cloud_region`.
* `xcor.integration.name`: Always `aws`.
* `aws_account_alias`: The alias configured for the account in the XCOR AWS
  integration, or else its AWS account alias. It's absent if the account has
  neither, or if the account's read role isn't allowed `iam:ListAccountAliases`,
  which read role template 1.2.0 grants. A permissions boundary or service
  control policy on the role must also allow it. It can also be absent for a
  while when XCOR can't read the AWS account alias, for example while IAM fails
  or throttles the request, and appears once a read succeeds. The label is part
  of each time series' identity, so an account with no alias configured gets new
  time series when its AWS account alias appears, changes, or goes away.
* The instance's resource tags, where XCOR reads them, such as `aws.tag.name`
  from the `Name` tag.

These time series don't have a `host` or `dimension_AutoScalingGroupName`
label. They don't have the `MetricName`, `instance` or `aws_exporter_arn`
labels that streamed metrics have, or the custom resource attributes you add
with stream parameters. The example drop rule in
[Control metric volume](#control-metric-volume) filters on `aws_exporter_arn`,
so it doesn't drop them.

### Amazon EC2 host status and instance age timestamps

XCOR reads every running instance in each account and region every two minutes
by default, and writes one sample of each metric per instance. Each sample's
timestamp is the time XCOR finished reading that account and region.
XCOR Support can set an interval from one to four minutes. A change EC2
reports in an instance's system status appears in XCOR at the next read.

### Query Amazon EC2 host status and instance age metrics

Each instance has a sample every read, so at the default interval instant
queries and range queries work as they do for streamed metrics. For example,
this query returns the instances whose system status is impaired:

```text theme={null}
aws_ec2_host_ok == 0
```

This query returns the instances that have run for more than 90 days since
their last launch, with each one's age in days:

```text theme={null}
aws_ec2_instance_age / 86400 > 90
```

The metrics have no `host` label, so a monitor converted from Datadog that
groups by `host` puts every matching instance in one group and alerts on their
average age. Group by `dimension_InstanceId` for one alert per instance.

The `xcor.integration.name` label and your AWS resource tag labels
(`aws.tag.<key>`) contain dots, so quote them in a selector:

```text theme={null}
aws_ec2_host_ok{"xcor.integration.name"="aws"}
```

### Validate Amazon EC2 host status and instance age ingestion

After XCOR Support turns on these reads, run this query. It counts the
running instances with a host status value in the last 10 minutes in each
account and region:

```text theme={null}
count by (cloud_account_id, cloud_region) (max by (cloud_account_id, cloud_region, dimension_InstanceId) (last_over_time(aws_ec2_host_ok[10m])))
```

For instance age, run:

```text theme={null}
count by (cloud_account_id, cloud_region) (max by (cloud_account_id, cloud_region, dimension_InstanceId) (last_over_time(aws_ec2_instance_age[10m])))
```

The queries read 10 minutes so that a single missed read doesn't hide an
instance. Each account and region with running instances should appear. Each
instance's newest value should be no older than one read interval, two minutes
by default, plus the time XCOR takes to read all your accounts and regions.

### Troubleshoot Amazon EC2 host status and instance age metrics

* **No EC2 host metrics for one account.** Check the account's read role
  version. A role older than 1.1.0 denies `ec2:DescribeInstances` and
  `ec2:DescribeInstanceStatus`.
* **`aws_ec2_instance_age` but no `aws_ec2_host_ok` for an account.** The role
  grants `ec2:DescribeInstances` but not `ec2:DescribeInstanceStatus`.
* **Gaps during busy periods.** EC2 limits each API action's request rate in
  each account and region, and the calls your own tools make count against the
  same limit. When EC2 throttles a read, XCOR writes what it read before the
  throttle and reads every instance again at the next read.
* **`aws_ec2_host_ok` is 1 while an instance status check fails.** Only an
  `impaired` system status reads 0. A new instance reads 1 while its status is
  `initializing`, and a failing instance status check doesn't change the metric.
* **An instance's age dropped to near 0.** The age is measured from the
  instance's last launch, which a stop and start resets.
* **An `aws_ec2_host_ok` time series has no `dimension_InstanceType` label.** The
  instance started during that read. The next read writes the label.

## Monitor drop rules

To view the rate at which the example CloudWatch Metric Streams drop rule drops
data points, use the following PromQL query:

```text theme={null}
sum(sum_per_second(chrono_metrics_drop_rule_dpps_matched{mode="enabled", applied="true", slug="drop-cloudwatch-metric-stream-metrics"}[5m]))
```


## Related topics

- [Amazon CloudWatch Logs and Metrics destination plugin](https://docs-xcor.paloaltonetworks.com/ingest/pipeline/plugins/destination-plugins/amazon-cloudwatch.md)
- [Cortex XCOR-managed dashboards](https://docs-xcor.paloaltonetworks.com/observe/dashboards/managed-dashboards.md)
- [Ingest metric and trace data](https://docs-xcor.paloaltonetworks.com/ingest/metrics-traces.md)
- [Integrate with Cortex XCOR](https://docs-xcor.paloaltonetworks.com/integrate.md)
- [Ingest AWS logs](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/server/aws/logs.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.