> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingest AWS logs

> Learn how to send AWS logs to Cortex XCOR.

Cortex XCOR receives Amazon Web Services (AWS) logs through
[Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html).
In each AWS account and region, a Firehose stream sends logs to the AWS logs ingest
endpoint in your XCOR tenant. CloudWatch Logs reach the stream through a
subscription, and some AWS services send their logs to the stream directly.

## What it collects

* CloudWatch Logs: Logs that AWS services and your applications write to CloudWatch
  log groups. Use an account-level subscription policy to cover every log group in a
  region, including new ones, or add a subscription filter to each log group.
  Subscriptions cover only log groups in the Standard log class. CloudTrail can also
  send its events to a log group.
* Logs sent straight to Firehose: Services such as Virtual Private Cloud (VPC)
  Flow Logs and Web Application Firewall (WAF) can deliver logs to a Firehose stream
  without a log group.
* Not collected: Logs that AWS writes only to Amazon S3, such as Classic Load
  Balancer access logs.

## How it works

The following diagram shows how AWS log sources reach XCOR and where Firehose
stores eligible failed deliveries.

```mermaid actions={false} theme={null}
graph LR
  accTitle: AWS log delivery
  accDescr: CloudWatch Logs, VPC Flow Logs, and AWS WAF send logs through Amazon Data Firehose to the Cortex XCOR AWS logs ingest endpoint. Eligible failed deliveries go to Amazon S3.
  subgraph AWS-account
    subgraph Region
      lg[CloudWatch
          log groups]
      vl[VPC Flow Logs,
          WAF]
      adf[Amazon Data
          Firehose]
      s3[S3 bucket for eligible
          failed data]
      lg -- subscription --> adf
      vl --> adf
      adf -. retryable failures .-> s3
    end
  end
  subgraph op[Cortex XCOR]
    subgraph Tenant
      ep[AWS logs
         ingest endpoint]
    end
  end
  adf --> ep

classDef greenFill stroke:#60E684,fill:#2FBF71,stroke-width:2px,color:#FFFFFF
class ep,op greenFill;

classDef blueFill fill:#000243,stroke-width:2px,color:#FFFFFF
class lg,vl,adf,s3,AWS-account blueFill;

classDef orange stroke-width:2px,color:#FFFFFF,stroke:#FF9D1C
class lg,vl,adf,s3,AWS-account orange
```

## Before you begin

* AWS permissions: The identity that runs the setup needs the permissions listed
  in [Required permissions](#required-permissions).
* An XCOR API token: Create or use an API token for a restricted service
  account with write-only permission. For more information, see
  [Create a restricted service account](https://docs-xcor.paloaltonetworks.com/administer/accounts-teams/service-accounts.md#create-a-restricted-service-account).

- Your organization name: The `<ORG_NAME>` segment in
  `https://<ORG_NAME>.chronosphere.io`.

* Existing subscriptions: An account can have one account-level subscription policy
  in each region, and each log group allows five subscription filters of its own,
  which the account-level policy doesn't count against. If another tool already
  uses the account-level policy, subscribe log groups individually. To inspect the
  account policies, run:

  ```shell theme={null}
  aws logs describe-account-policies --policy-type SUBSCRIPTION_FILTER_POLICY --region <REGION>
  ```

### Required permissions

To run the setup using the
[AWS provider for Terraform](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)
or CloudFormation in one account, the identity that runs it needs the following
permissions. These permissions allow the identity to create and remove the setup
resources. StackSets and the console need additional permissions.

* Amazon Data Firehose
  * Actions: `CreateDeliveryStream`, `DeleteDeliveryStream`,
    `DescribeDeliveryStream`, `UpdateDestination`, `ListTagsForDeliveryStream`,
    `TagDeliveryStream`, `UntagDeliveryStream`
  * Resource: The stream
* Amazon S3
  * Actions: `CreateBucket`, `DeleteBucket`, `PutLifecycleConfiguration`,
    `PutBucketTagging`, `ListBucket`, `Get*`, and `DeleteObject` to empty the bucket
    when you remove it
  * Resources: The failed-data bucket and its objects
* CloudWatch Logs
  * Actions: `CreateLogGroup`, `DeleteLogGroup`, `PutRetentionPolicy`,
    `CreateLogStream`, `DeleteLogStream`, `DescribeLogGroups`,
    `DescribeLogStreams`, `ListTagsForResource`, `TagResource`, `PutAccountPolicy`,
    `DeleteAccountPolicy`, `DescribeAccountPolicies`, `PutSubscriptionFilter`,
    `DeleteSubscriptionFilter`, `DescribeSubscriptionFilters`
  * Resource: All resources, because account policies have no resource ARN
* IAM
  * Actions: `CreateRole`, `DeleteRole`, `GetRole`, `PutRolePolicy`,
    `GetRolePolicy`, `DeleteRolePolicy`, `ListRolePolicies`,
    `ListAttachedRolePolicies`, `ListInstanceProfilesForRole`, `PassRole`
  * Resources: The two roles the setup creates
* AWS Secrets Manager
  * Actions: `CreateSecret`, `DeleteSecret`, `DescribeSecret`, `UpdateSecret`,
    `GetSecretValue`, `TagResource`, `UntagResource`
  * Resource: The token secret
* CloudFormation, for the CloudFormation tab only
  * Actions: `CreateChangeSet`, `DescribeChangeSet`, `ExecuteChangeSet`,
    `DeleteChangeSet`, `DescribeStacks`, `DescribeStackEvents`, `DeleteStack`,
    `GetTemplateSummary`
  * Resource: The stack

The setup creates two roles:

* Firehose role: Trusted by Firehose in your account. It can write to the
  failed-data bucket, read the token secret, and write the stream's error log.
* Subscription role: Trusted by CloudWatch Logs in your account. It can put
  records into the stream.

If the token secret uses a customer managed AWS Key Management Service key, add
`kms:Decrypt` permission on the key to the Firehose role, and grant the role access
in the key policy.

The setup has no access to your existing buckets or other resources, and it doesn't
send logs written before it's in place.

## Firehose settings

Use these settings regardless of how you create the stream:

Firehose bills each CloudWatch Logs record at its compressed size, rounded up to
5 KB. Log groups that send few events per batch cost more per log. Additional AWS
data transfer charges can apply. Firehose sends a failed batch again on each retry,
so data transfer can grow while deliveries fail.

* Source: `Direct PUT`.
* Destination: `HTTP Endpoint`.
* HTTP endpoint URL:
  `https://<ORG_NAME>.chronosphere.io/api/v1/data/logs/ingest/integrations/aws`.
* Authentication: An AWS Secrets Manager secret containing
  `{"api_key":"<API_TOKEN>"}`. Firehose picks up a rotated token within about
  10 minutes. Using the API token as the endpoint access key also works, but stores
  it in the stream settings.
* Content encoding: `GZIP` for smaller requests and lower data transfer costs.
* Retry duration: `7200` seconds, the longest Firehose allows. Records that still
  fail go to the S3 bucket.
* Buffer size: `1` MiB. Keep this value at 4 MiB or less. Firehose treats the size
  as a hint and encodes each record in a text representation larger than the source
  record. XCOR rejects requests over 16 MiB with a `413`, and Firehose drops a
  batch that gets a `413` without retrying it or backing it up.
* Buffer interval: `60` seconds.
* Decompression and message extraction: Off. XCOR decodes CloudWatch Logs
  records. Message extraction removes the log group and log stream used to identify
  the service.
* Source record backup in Amazon S3: `Failed data only`.
* Backup bucket lifecycle: Expire objects after `14` days. XCOR drops
  retried logs more than 24 hours old.
* Parameters: None. For the exception for VPC Flow Logs in a custom format, see
  [Send logs straight to Firehose](#send-logs-straight-to-firehose).

## Set up log delivery

Set up delivery in each account and region you want logs from.
The Integrations Hub has no configuration form for AWS logs. Configure delivery
in AWS by using one of the following methods.

> [tab] Terraform
>
> 1. Store your API token in AWS Secrets Manager in the region you're setting up. Note
>    the secret's ARN.
>
>    Read the token without echoing it, so it stays out of your shell history. Run the
>    `read` command on its own:
>
>    ```shell theme={null}
>    printf 'API token: '; read -rs API_TOKEN; echo
>    ```
>
>    Then create the secret:
>
>    ```shell theme={null}
>    if [ -n "$API_TOKEN" ]; then
>      printf '{"api_key":"%s"}' "$API_TOKEN" |
>        aws secretsmanager create-secret --region <REGION> \
>          --name cortex-xcor/aws-logs-api-token \
>          --secret-string file:///dev/stdin
>    else
>      echo 'API_TOKEN is empty: run the read command first'
>    fi
>    unset API_TOKEN
>    ```
>
> 2. The configuration was tested with version 6 of the `aws` provider. The
>    configuration also requires the `time` provider. Set the region on the `aws`
>    provider:
>
>    ```terraform theme={null}
>    terraform {
>      required_providers {
>        aws  = { source = "hashicorp/aws", version = ">= 6.0" }
>        time = { source = "hashicorp/time" }
>      }
>    }
>
>    provider "aws" {
>      region = "<REGION>"
>    }
>    ```
>
> 3. Set values for these variables, and apply the configuration in each account and
>    region. It creates the Firehose stream, S3 bucket for failed data with a lifecycle
>    rule, and IAM roles. Unless you turn it off, it also creates an account-level
>    subscription policy that sends every log group in the region except the stream's
>    own error logs.
>
>    ```terraform expandable Terraform example icon="square-terminal" theme={null}
>    # Variables
>
>    variable "xcor_org_name" {
>      type        = string
>      description = "The name of your Cortex XCOR organization, which is the subdomain name before .chronosphere.io."
>    }
>
>    variable "api_token_secret_arn" {
>      type        = string
>      description = "The ARN of the Secrets Manager secret holding {\"api_key\":\"<API_TOKEN>\"}, in the stream's region."
>    }
>
>    variable "stream_name" {
>      type        = string
>      description = "The name of the Firehose stream."
>      default     = "cortex-xcor-logs"
>    }
>
>    variable "create_account_policy" {
>      type        = bool
>      description = "Set to false if this account and region already has an account-level subscription policy."
>      default     = true
>    }
>
>    variable "failed_data_expiration_days" {
>      type        = number
>      description = "Days to keep records Firehose couldn't deliver."
>      default     = 14
>    }
>
>    locals {
>      tags = {
>        "cortex-xcor:template"         = "aws-logs-firehose"
>        "cortex-xcor:template-version" = "1.0.0"
>      }
>    }
>
>    data "aws_caller_identity" "current" {}
>
>    # S3 bucket and log group for failed deliveries
>
>    resource "aws_s3_bucket" "failed_data" {
>      bucket_prefix = "cortex-xcor-logs-failed-"
>      tags          = local.tags
>    }
>
>    resource "aws_s3_bucket_lifecycle_configuration" "failed_data" {
>      bucket = aws_s3_bucket.failed_data.id
>      rule {
>        id     = "expire-failed-data"
>        status = "Enabled"
>        filter {}
>        expiration {
>          days = var.failed_data_expiration_days
>        }
>      }
>    }
>
>    resource "aws_cloudwatch_log_group" "firehose" {
>      name              = "/aws/kinesisfirehose/${var.stream_name}"
>      retention_in_days = 14
>    }
>
>    resource "aws_cloudwatch_log_stream" "firehose" {
>      name           = "DestinationDelivery"
>      log_group_name = aws_cloudwatch_log_group.firehose.name
>    }
>
>    # Firehose stream
>
>    data "aws_iam_policy_document" "firehose_trust" {
>      statement {
>        actions = ["sts:AssumeRole"]
>        principals {
>          type        = "Service"
>          identifiers = ["firehose.amazonaws.com"]
>        }
>        condition {
>          test     = "StringEquals"
>          variable = "aws:SourceAccount"
>          values   = [data.aws_caller_identity.current.account_id]
>        }
>      }
>    }
>
>    data "aws_iam_policy_document" "firehose_access" {
>      statement {
>        actions = [
>          "s3:AbortMultipartUpload",
>          "s3:GetBucketLocation",
>          "s3:GetObject",
>          "s3:ListBucket",
>          "s3:ListBucketMultipartUploads",
>          "s3:PutObject",
>        ]
>        resources = [aws_s3_bucket.failed_data.arn, "${aws_s3_bucket.failed_data.arn}/*"]
>      }
>      statement {
>        actions   = ["secretsmanager:GetSecretValue"]
>        resources = [var.api_token_secret_arn]
>      }
>      statement {
>        actions   = ["logs:PutLogEvents"]
>        resources = ["${aws_cloudwatch_log_group.firehose.arn}:*"]
>      }
>    }
>
>    resource "aws_iam_role" "firehose" {
>      name_prefix        = "cortex-xcor-logs-firehose-"
>      assume_role_policy = data.aws_iam_policy_document.firehose_trust.json
>    }
>
>    resource "aws_iam_role_policy" "firehose" {
>      role   = aws_iam_role.firehose.id
>      policy = data.aws_iam_policy_document.firehose_access.json
>    }
>
>    resource "aws_kinesis_firehose_delivery_stream" "logs" {
>      name        = var.stream_name
>      destination = "http_endpoint"
>      tags        = local.tags
>
>      http_endpoint_configuration {
>        name               = "Cortex XCOR"
>        url                = "https://${var.xcor_org_name}.chronosphere.io/api/v1/data/logs/ingest/integrations/aws"
>        buffering_size     = 1    # MiB
>        buffering_interval = 60   # seconds
>        retry_duration     = 7200 # seconds
>        role_arn           = aws_iam_role.firehose.arn
>        s3_backup_mode     = "FailedDataOnly"
>
>        secrets_manager_configuration {
>          enabled    = true
>          secret_arn = var.api_token_secret_arn
>          role_arn   = aws_iam_role.firehose.arn
>        }
>
>        s3_configuration {
>          role_arn           = aws_iam_role.firehose.arn
>          bucket_arn         = aws_s3_bucket.failed_data.arn
>          buffering_size     = 10  # MiB
>          buffering_interval = 300 # seconds
>          compression_format = "GZIP"
>        }
>
>        request_configuration {
>          content_encoding = "GZIP"
>        }
>
>        cloudwatch_logging_options {
>          enabled         = true
>          log_group_name  = aws_cloudwatch_log_group.firehose.name
>          log_stream_name = aws_cloudwatch_log_stream.firehose.name
>        }
>      }
>    }
>
>    # CloudWatch Logs subscription
>
>    data "aws_iam_policy_document" "subscription_trust" {
>      statement {
>        actions = ["sts:AssumeRole"]
>        principals {
>          type        = "Service"
>          identifiers = ["logs.amazonaws.com"]
>        }
>        condition {
>          test     = "StringLike"
>          variable = "aws:SourceArn"
>          values   = ["arn:aws:logs:*:${data.aws_caller_identity.current.account_id}:*"]
>        }
>      }
>    }
>
>    data "aws_iam_policy_document" "subscription_access" {
>      statement {
>        actions   = ["firehose:PutRecord", "firehose:PutRecordBatch"]
>        resources = [aws_kinesis_firehose_delivery_stream.logs.arn]
>      }
>    }
>
>    resource "aws_iam_role" "subscription" {
>      name_prefix        = "cortex-xcor-logs-subscription-"
>      assume_role_policy = data.aws_iam_policy_document.subscription_trust.json
>    }
>
>    resource "aws_iam_role_policy" "subscription" {
>      role   = aws_iam_role.subscription.id
>      policy = data.aws_iam_policy_document.subscription_access.json
>    }
>
>    # CloudWatch Logs sends a test record when the policy is created, and a new
>    # role's permissions can take a few seconds to take effect.
>    resource "time_sleep" "subscription_role" {
>      depends_on      = [aws_iam_role_policy.subscription]
>      create_duration = "30s"
>    }
>
>    resource "aws_cloudwatch_log_account_policy" "logs" {
>      count      = var.create_account_policy ? 1 : 0
>      depends_on = [time_sleep.subscription_role]
>
>      policy_name = "cortex-xcor-logs"
>      policy_type = "SUBSCRIPTION_FILTER_POLICY"
>      scope       = "ALL"
>      # The stream's own error logs are excluded, or they'd loop back into it.
>      selection_criteria = "LogGroupName NOT IN [\"${aws_cloudwatch_log_group.firehose.name}\"]"
>      policy_document = jsonencode({
>        DestinationArn = aws_kinesis_firehose_delivery_stream.logs.arn
>        RoleArn        = aws_iam_role.subscription.arn
>        FilterPattern  = ""
>      })
>    }
>
>    # Used to subscribe log groups individually
>
>    output "stream_arn" {
>      value = aws_kinesis_firehose_delivery_stream.logs.arn
>    }
>
>    output "subscription_role_arn" {
>      value = aws_iam_role.subscription.arn
>    }
>    ```
>
> 4. If you set `create_account_policy = false`, follow
>    [Subscribe individual log groups](#subscribe-individual-log-groups) with the
>    `stream_arn` and `subscription_role_arn` outputs.

> [tab] AWS Management Console
>
> 1. In [AWS Secrets Manager](https://console.aws.amazon.com/secretsmanager/), store a
>    secret of type **Other type of secret** with the key `api_key` and your API token
>    as the value. Name it `cortex-xcor/aws-logs-api-token`.
> 2. Open the
>    [Amazon Data Firehose console](https://console.aws.amazon.com/firehose/) and select
>    **Create Firehose stream**.
> 3. Set the following options:
>    * **Source:** `Direct PUT`
>    * **Destination:** `HTTP Endpoint`
>    * **Firehose stream name:** `cortex-xcor-logs`
>    * **Transform records:** If the console offers **Decompress source records from
>      Amazon CloudWatch Logs**, leave decompression and message extraction off.
>    * **Destination settings:**
>      * **HTTP endpoint name:** `Cortex XCOR`
>      * **HTTP endpoint URL:**
>        `https://<ORG_NAME>.chronosphere.io/api/v1/data/logs/ingest/integrations/aws`.
>        Replace `<ORG_NAME>` with your organization name.
>      * **Authentication:** `Use AWS Secrets Manager`, and select the secret you created.
>      * **Content encoding:** `GZIP`
>      * **Retry duration:** `7200 seconds`
>      * Expand **Buffer hints**.
>        * **Buffer size:** `1 MiB`
>        * **Buffer interval:** `60 seconds`
>    * **Backup settings:**
>      * **Source record backup in Amazon S3:** `Failed data only`
>      * **S3 backup bucket:** Create a bucket for this stream with default settings. In
>        the bucket's **Management** tab, add a lifecycle rule that expires objects
>        after 14 days. Don't add that rule to a bucket that holds other data.
>      * Expand **Buffer hints, compression and encryption**, and set
>        **Compression for data records** to `GZIP`.
>    * **Advanced settings:** Under **Service access**, let the console create an IAM
>      role. Confirm that the role allows `secretsmanager:GetSecretValue` on the secret.
> 4. Select **Create Firehose stream**.
> 5. In the [IAM console](https://console.aws.amazon.com/iam/), create a role that
>    CloudWatch Logs can assume. Set the trusted entity to `logs.amazonaws.com`, with
>    the condition that `aws:SourceArn` is like
>    `arn:aws:logs:<REGION>:<ACCOUNT_ID>:*`. Grant the role `firehose:PutRecord` and
>    `firehose:PutRecordBatch` on the stream.
> 6. Use the AWS Command Line Interface to subscribe log groups with an
>    account-level subscription policy. Exclude the stream's own error log group to
>    prevent its logs from looping back into the stream:
>
>    ```shell theme={null}
>    aws logs put-account-policy --region <REGION> \
>      --policy-name cortex-xcor-logs \
>      --policy-type SUBSCRIPTION_FILTER_POLICY \
>      --scope ALL \
>      --selection-criteria 'LogGroupName NOT IN ["/aws/kinesisfirehose/cortex-xcor-logs"]' \
>      --policy-document '{"DestinationArn":"<STREAM_ARN>","RoleArn":"<ROLE_ARN>","FilterPattern":""}'
>    ```
>
>    If the account already has an account-level subscription policy, subscribe log
>    groups individually.

> [tab] CloudFormation
>
> 1. Store your API token in AWS Secrets Manager in the region you're setting up. Note
>    the secret's ARN.
>
>    Read the token without echoing it, so it stays out of your shell history. Run the
>    `read` command on its own:
>
>    ```shell theme={null}
>    printf 'API token: '; read -rs API_TOKEN; echo
>    ```
>
>    Then create the secret:
>
>    ```shell theme={null}
>    if [ -n "$API_TOKEN" ]; then
>      printf '{"api_key":"%s"}' "$API_TOKEN" |
>        aws secretsmanager create-secret --region <REGION> \
>          --name cortex-xcor/aws-logs-api-token \
>          --secret-string file:///dev/stdin
>    else
>      echo 'API_TOKEN is empty: run the read command first'
>    fi
>    unset API_TOKEN
>    ```
>
> 2. Save this template as `cortex-xcor-aws-logs.yaml`. It creates the Firehose stream,
>    S3 bucket for failed data with a lifecycle rule, and IAM roles. Unless you turn it
>    off, it also creates an account-level subscription policy that sends every log
>    group in the region except the stream's own error logs.
>
>    ```yaml expandable CloudFormation template icon="file-code" theme={null}
>    AWSTemplateFormatVersion: "2010-09-09"
>    Description: Sends CloudWatch Logs in this account and region to Cortex XCOR through Amazon Data Firehose. Template version 1.1.0.
>
>    Parameters:
>      OrgName:
>        Type: String
>        Description: Your Cortex XCOR organization name, the subdomain before .chronosphere.io.
>      ApiTokenSecretArn:
>        Type: String
>        Default: ""
>        Description: ARN of the Secrets Manager secret holding {"api_key":"<API_TOKEN>"}, in this stack's region. Leave empty to set ApiToken instead.
>      ApiToken:
>        Type: String
>        Default: ""
>        NoEcho: true
>        Description: The API token, for a stack that creates its own secret, such as a StackSet instance. Leave empty to set ApiTokenSecretArn instead.
>      StreamName:
>        Type: String
>        Default: cortex-xcor-logs
>      CreateAccountPolicy:
>        Type: String
>        Default: "true"
>        AllowedValues: ["true", "false"]
>        Description: Set to false if this account and region already has an account-level subscription policy.
>      FailedDataExpirationDays:
>        Type: Number
>        Default: 14
>        MinValue: 1
>        Description: Days to keep records Firehose couldn't deliver.
>
>    Rules:
>      OneTokenSource:
>        Assertions:
>          - Assert: !Or [!Not [!Equals [!Ref ApiTokenSecretArn, ""]], !Not [!Equals [!Ref ApiToken, ""]]]
>            AssertDescription: Set ApiTokenSecretArn or ApiToken.
>          - Assert: !Or [!Equals [!Ref ApiTokenSecretArn, ""], !Equals [!Ref ApiToken, ""]]
>            AssertDescription: Set only one of ApiTokenSecretArn and ApiToken.
>
>    Conditions:
>      CreatePolicy: !Equals [!Ref CreateAccountPolicy, "true"]
>      CreateSecret: !Equals [!Ref ApiTokenSecretArn, ""]
>
>    Resources:
>      ApiTokenSecret:
>        Type: AWS::SecretsManager::Secret
>        Condition: CreateSecret
>        Properties:
>          Description: Cortex XCOR API token for the AWS logs Firehose stream
>          SecretString: !Sub '{"api_key":"${ApiToken}"}'
>
>      FailedDataBucket:
>        Type: AWS::S3::Bucket
>        DeletionPolicy: RetainExceptOnCreate
>        UpdateReplacePolicy: Retain
>        Properties:
>          LifecycleConfiguration:
>            Rules:
>              - Id: expire-failed-data
>                Status: Enabled
>                ExpirationInDays: !Ref FailedDataExpirationDays
>          Tags:
>            - { Key: "cortex-xcor:template", Value: aws-logs-firehose }
>            - { Key: "cortex-xcor:template-version", Value: "1.1.0" }
>
>      FirehoseLogGroup:
>        Type: AWS::Logs::LogGroup
>        Properties:
>          LogGroupName: !Sub /aws/kinesisfirehose/${StreamName}
>          RetentionInDays: 14
>
>      FirehoseLogStream:
>        Type: AWS::Logs::LogStream
>        Properties:
>          LogGroupName: !Ref FirehoseLogGroup
>          LogStreamName: DestinationDelivery
>
>      FirehoseRole:
>        Type: AWS::IAM::Role
>        Properties:
>          AssumeRolePolicyDocument:
>            Version: "2012-10-17"
>            Statement:
>              - Effect: Allow
>                Principal: { Service: firehose.amazonaws.com }
>                Action: sts:AssumeRole
>                Condition:
>                  StringEquals:
>                    aws:SourceAccount: !Ref AWS::AccountId
>          Policies:
>            - PolicyName: delivery
>              PolicyDocument:
>                Version: "2012-10-17"
>                Statement:
>                  - Effect: Allow
>                    Action: [s3:AbortMultipartUpload, s3:GetBucketLocation, s3:GetObject, s3:ListBucket, s3:ListBucketMultipartUploads, s3:PutObject]
>                    Resource: [!GetAtt FailedDataBucket.Arn, !Sub "${FailedDataBucket.Arn}/*"]
>                  - Effect: Allow
>                    Action: secretsmanager:GetSecretValue
>                    Resource: !If [CreateSecret, !Ref ApiTokenSecret, !Ref ApiTokenSecretArn]
>                  - Effect: Allow
>                    Action: logs:PutLogEvents
>                    Resource: !GetAtt FirehoseLogGroup.Arn
>
>      DeliveryStream:
>        Type: AWS::KinesisFirehose::DeliveryStream
>        Properties:
>          DeliveryStreamName: !Ref StreamName
>          DeliveryStreamType: DirectPut
>          Tags:
>            - { Key: "cortex-xcor:template", Value: aws-logs-firehose }
>            - { Key: "cortex-xcor:template-version", Value: "1.1.0" }
>          HttpEndpointDestinationConfiguration:
>            EndpointConfiguration:
>              Name: Cortex XCOR
>              Url: !Sub https://${OrgName}.chronosphere.io/api/v1/data/logs/ingest/integrations/aws
>            SecretsManagerConfiguration:
>              Enabled: true
>              SecretARN: !If [CreateSecret, !Ref ApiTokenSecret, !Ref ApiTokenSecretArn]
>              RoleARN: !GetAtt FirehoseRole.Arn
>            RequestConfiguration:
>              ContentEncoding: GZIP
>            BufferingHints:
>              SizeInMBs: 1
>              IntervalInSeconds: 60
>            RetryOptions:
>              DurationInSeconds: 7200
>            RoleARN: !GetAtt FirehoseRole.Arn
>            S3BackupMode: FailedDataOnly
>            S3Configuration:
>              BucketARN: !GetAtt FailedDataBucket.Arn
>              RoleARN: !GetAtt FirehoseRole.Arn
>              CompressionFormat: GZIP
>              BufferingHints:
>                SizeInMBs: 10
>                IntervalInSeconds: 300
>            CloudWatchLoggingOptions:
>              Enabled: true
>              LogGroupName: !Ref FirehoseLogGroup
>              LogStreamName: !Ref FirehoseLogStream
>
>      SubscriptionRole:
>        Type: AWS::IAM::Role
>        Properties:
>          AssumeRolePolicyDocument:
>            Version: "2012-10-17"
>            Statement:
>              - Effect: Allow
>                Principal: { Service: logs.amazonaws.com }
>                Action: sts:AssumeRole
>                Condition:
>                  StringLike: { "aws:SourceArn": !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:*" }
>          Policies:
>            - PolicyName: put-to-firehose
>              PolicyDocument:
>                Version: "2012-10-17"
>                Statement:
>                  - Effect: Allow
>                    Action: [firehose:PutRecord, firehose:PutRecordBatch]
>                    Resource: !GetAtt DeliveryStream.Arn
>
>      AccountPolicy:
>        Type: AWS::Logs::AccountPolicy
>        Condition: CreatePolicy
>        Properties:
>          PolicyName: cortex-xcor-logs
>          PolicyType: SUBSCRIPTION_FILTER_POLICY
>          Scope: ALL
>          # The stream's own error logs are excluded, or they'd loop back into it.
>          SelectionCriteria: !Sub 'LogGroupName NOT IN ["/aws/kinesisfirehose/${StreamName}"]'
>          PolicyDocument: !Sub '{"DestinationArn":"${DeliveryStream.Arn}","RoleArn":"${SubscriptionRole.Arn}","FilterPattern":""}'
>
>    Outputs:
>      DeliveryStreamArn:
>        Value: !GetAtt DeliveryStream.Arn
>      SubscriptionRoleArn:
>        Value: !GetAtt SubscriptionRole.Arn
>    ```
>
> 3. Deploy the stack:
>
>    ```shell theme={null}
>    aws cloudformation deploy --region <REGION> \
>      --template-file cortex-xcor-aws-logs.yaml \
>      --stack-name cortex-xcor-aws-logs \
>      --capabilities CAPABILITY_IAM \
>      --parameter-overrides OrgName=<ORG_NAME> ApiTokenSecretArn=<SECRET_ARN>
>    ```
>
>    `deploy` keeps a parameter's previous value when you leave it out. To switch an
>    existing stack between `ApiToken` and `ApiTokenSecretArn`, pass the one you stop
>    using as empty, such as `ApiToken=""`.
>
> 4. If you set `CreateAccountPolicy=false`, follow
>    [Subscribe individual log groups](#subscribe-individual-log-groups) with the
>    `DeliveryStreamArn` and `SubscriptionRoleArn` stack outputs.

### Set up many accounts and regions

Set up one stream in each account and region you want logs from. Each stream reports
its own deliveries, so a problem appears in the account and region it affects.

To deploy to many accounts in an organization in AWS Organizations, run the
CloudFormation template as a StackSet with service-managed permissions. Pass the API
token as `ApiToken` instead of `ApiTokenSecretArn`. Each stack instance then creates
its own secret, so accounts added later through automatic deployment need no extra setup,
unless they already have an account-level subscription policy. CloudFormation shows
`ApiToken` as `****`, but keeps it as a StackSet parameter. If the token should live
only in Secrets Manager, set up each account with `ApiTokenSecretArn` instead.

A StackSet exists only in the region where you create it. Choose one region to
administer it from, and pass that region as `--region <ADMIN_REGION>` on every
StackSet command, including later rotations and removal. With a delegated
administrator, pick one of the
[regions AWS supports for delegated administrators](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-orgs-delegated-admin.html).

Before you start,
[enable all features](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_org_support-all-features.html)
in AWS Organizations and
[activate trusted access](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-orgs-activate-trusted-access.html)
for StackSets from the organization's management account:

```shell theme={null}
aws cloudformation activate-organizations-access
```

If a target account already has the single-account setup from this page in a target
region, remove it first, as described in [Remove log delivery](#remove-log-delivery).
The StackSet's stacks use the same stream and log group names, so their instances fail
there. The management account isn't targeted, so keep its setup.

To administer the StackSet from a
[delegated administrator](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-orgs-delegated-admin.html)
account instead, register that account from the management account:

```shell theme={null}
aws organizations register-delegated-administrator \
  --service-principal member.org.stacksets.cloudformation.amazonaws.com \
  --account-id <ADMIN_ACCOUNT_ID>
```

From a delegated administrator, add `--call-as DELEGATED_ADMIN` to every StackSet
command on this page: `create-stack-set`, `create-stack-instances`,
`update-stack-instances`, `update-stack-set`, `describe-stack-set-operation`,
`list-stack-instances`, `delete-stack-instances`, and `delete-stack-set`. The
delegated administrator also needs `organizations:ListDelegatedAdministrators`.

The following commands read the API token with `read -rs`. This command works in `bash`
and `zsh` and doesn't show the token as you paste it. Run the `read` command on its
own, before the commands that use the token.

1. Read the API token:

   ```shell theme={null}
   printf 'API token: '; read -rs API_TOKEN; echo
   ```

2. Write the parameters to a file only you can read, so the token stays out of your
   shell history:

   ```shell theme={null}
   rm -f stackset-params.json
   if [ -n "$API_TOKEN" ]; then
     (umask 077 && cat > stackset-params.json) <<EOF
   [
     {"ParameterKey": "OrgName", "ParameterValue": "<ORG_NAME>"},
     {"ParameterKey": "ApiToken", "ParameterValue": "$API_TOKEN"}
   ]
   EOF
   else
     echo 'API_TOKEN is empty: run the read command first'
   fi
   unset API_TOKEN
   ```

3. Create the StackSet from the template in the CloudFormation tab, saved as
   `cortex-xcor-aws-logs.yaml`, and then delete the parameters file:

   ```shell theme={null}
   if [ -f stackset-params.json ]; then
     aws cloudformation create-stack-set --region <ADMIN_REGION> \
       --stack-set-name cortex-xcor-aws-logs \
       --template-body file://cortex-xcor-aws-logs.yaml \
       --permission-model SERVICE_MANAGED \
       --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \
       --capabilities CAPABILITY_IAM \
       --parameters file://stackset-params.json
     rm -f stackset-params.json
   else
     echo 'stackset-params.json is missing: run the previous step first'
   fi
   ```

   With automatic deployment, an account that joins a target organizational unit gets
   a stack, and an account that leaves has its stack deleted. The deleted stack's
   failed-data bucket is kept.

4. Deploy the StackSet to your organizational units and regions. `<OU_ID>` can be the
   organization's root ID (`r-...`) to cover every account.

   ```shell theme={null}
   OP_ID=$(aws cloudformation create-stack-instances --region <ADMIN_REGION> \
     --stack-set-name cortex-xcor-aws-logs \
     --deployment-targets OrganizationalUnitIds=<OU_ID> \
     --regions <REGION> <REGION> \
     --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \
     --query OperationId --output text)
   ```

   These preferences deploy to every account and region at once and keep going past
   accounts that fail. Without them, CloudFormation deploys to one account at a time
   and stops at the first failure, which also skips every later region. The
   preferences don't apply to automatic deployments.

5. Wait for the operation to finish, and then list the stack instances that failed.
   With these preferences, the operation reports `SUCCEEDED` even when some instances
   failed.

   ```shell theme={null}
   while :; do
     STATUS=$(aws cloudformation describe-stack-set-operation --region <ADMIN_REGION> \
       --stack-set-name cortex-xcor-aws-logs --operation-id "$OP_ID" \
       --query StackSetOperation.Status --output text)
     case "$STATUS" in RUNNING|QUEUED|STOPPING) sleep 30 ;; *) break ;; esac
   done
   echo "Operation $STATUS"
   aws cloudformation list-stack-instances --region <ADMIN_REGION> \
     --stack-set-name cortex-xcor-aws-logs \
     --filters Name=DETAILED_STATUS,Values=FAILED
   ```

A stack instance fails if its account already has an account-level subscription policy
in that region, because an account can have only one in each region. For each of those,
override `CreateAccountPolicy` to `false`, wait for the operation with the
commands in step 5, and then subscribe log groups individually in that account with
the `DeliveryStreamArn` and `SubscriptionRoleArn` outputs of its stack:

```shell theme={null}
OP_ID=$(aws cloudformation update-stack-instances --region <ADMIN_REGION> \
  --stack-set-name cortex-xcor-aws-logs \
  --deployment-targets Accounts=<ACCOUNT_ID> \
  --regions <REGION> \
  --parameter-overrides ParameterKey=CreateAccountPolicy,ParameterValue=false \
  --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \
  --query OperationId --output text)
```

Target accounts rather than an organizational unit, because an override set on an
organizational unit doesn't apply to accounts that join that unit later. Fix failed
instances promptly. Every later update to the StackSet retries creating them, which
makes the update much slower.

A StackSet with service-managed permissions doesn't deploy to the management account.
To collect that account's logs, set it up on its own as described earlier on this
page.

To rotate the token, read the new token on its own:

```shell theme={null}
printf 'API token: '; read -rs API_TOKEN; echo
```

Then update the StackSet's `ApiToken` parameter, keeping the other parameters:

```shell theme={null}
rm -f stackset-params.json
if [ -n "$API_TOKEN" ]; then
  (umask 077 && cat > stackset-params.json) <<EOF
[
  {"ParameterKey": "OrgName", "UsePreviousValue": true},
  {"ParameterKey": "StreamName", "UsePreviousValue": true},
  {"ParameterKey": "CreateAccountPolicy", "UsePreviousValue": true},
  {"ParameterKey": "FailedDataExpirationDays", "UsePreviousValue": true},
  {"ParameterKey": "ApiTokenSecretArn", "UsePreviousValue": true},
  {"ParameterKey": "ApiToken", "ParameterValue": "$API_TOKEN"}
]
EOF
  OP_ID=$(aws cloudformation update-stack-set --region <ADMIN_REGION> \
    --stack-set-name cortex-xcor-aws-logs \
    --use-previous-template \
    --capabilities CAPABILITY_IAM \
    --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \
    --parameters file://stackset-params.json \
    --query OperationId --output text)
  rm -f stackset-params.json
else
  echo 'API_TOKEN is empty: run the read command first'
fi
unset API_TOKEN
```

Wait for the operation and list failed instances with the commands in step 5. If any
instance is listed, fix its cause and run the update again to retry the instances that
didn't update. If CloudFormation reports that another operation is in
progress, such as an automatic deployment, wait for it to finish and run the update
again.

If the management account's own setup uses the same token, update its secret in each
region where you set it up. Read the new token again:

```shell theme={null}
printf 'API token: '; read -rs API_TOKEN; echo
```

Then update each region:

```shell theme={null}
if [ -n "$API_TOKEN" ]; then
  for region in <REGION> <REGION>; do
    printf '{"api_key":"%s"}' "$API_TOKEN" |
      aws secretsmanager put-secret-value --region "$region" \
        --secret-id cortex-xcor/aws-logs-api-token \
        --secret-string file:///dev/stdin
  done
else
  echo 'API_TOKEN is empty: run the read command first'
fi
unset API_TOKEN
```

Each Firehose stream starts using the new token within 10 minutes of its secret
changing. Don't revoke the old token until every stack and the management account's
secret have updated, and 10 more minutes have passed.

This page doesn't cover sending logs from many accounts to one stream.

### Subscribe individual log groups

To send only some log groups, or when another tool holds the account-level policy,
add a subscription filter to each log group:

```shell theme={null}
aws logs put-subscription-filter --region <REGION> \
  --log-group-name <LOG_GROUP> \
  --filter-name cortex-xcor-logs \
  --filter-pattern "" \
  --destination-arn <STREAM_ARN> \
  --role-arn <ROLE_ARN>
```

Don't include these log groups in the account-level policy, or their logs arrive
twice.

### Send logs straight to Firehose

Some services deliver logs to a Firehose stream without a log group. Give each
service its own stream, created in the console with the same settings under a unique
name. By default, both the Terraform configuration, which uses the
[AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs),
and the CloudFormation configuration create the account subscription policy, so
don't run them again for these streams. The account-level policy sends each
additional stream's error log group, `/aws/kinesisfirehose/<STREAM_NAME>`, to the
stream targeted by the policy unless you add the error log group to the policy
exclusion.

XCOR recognizes these logs on its own:

* AWS WAF: By the stream name, which AWS requires to start with
  `aws-waf-logs-`, and by the log format.
* Amazon API Gateway: REST API access logs, identified by a stream name that starts
  with `amazon-apigateway-`.
* VPC Flow Logs: By the default log format. For a custom format, add the stream
  parameter `xcor_integration` with the value `aws_vpc`.

For other logs sent straight to Firehose, set the `xcor_integration` parameter to a
recognized AWS integration ID, such as `aws_cloudtrail`, `aws_eks`, `aws_lambda`, or
`aws_rds`. XCOR ignores unknown values. Logs it can't otherwise recognize are
stored with the label `aws_other`.

## Validate log ingestion

After setup, allow at least one minute for the configured Firehose buffer interval
before validating ingestion.

1. In the Amazon Data Firehose console, open the stream. **HTTP endpoint delivery
   success** should report 100%, and **Records delivered to HTTP endpoint** should be
   non-zero. A successful delivery means XCOR accepted the request. It doesn't
   confirm that every log in the request passed ingestion validation.
2. In the navigation menu, select
   **Explorers > [Logs Explorer](https://docs-xcor.paloaltonetworks.com/investigate/querying/query-logs)**
   and run:

   ```text theme={null}
   ["xcor.integration.name"] = "aws" | summarize count() by xcor_integration
   ```

   Each AWS service sending logs appears as a row.
   For more about queries, see
   [Search and filter log data](https://docs-xcor.paloaltonetworks.com/investigate/querying/query-logs).

## Log labels and fields

Every log gets two integration labels:

* `xcor.integration.name`: always `aws`, the integration's catalog ID.
* `xcor_integration`: the AWS service that wrote the log, such as `aws_lambda`,
  `aws_rds`, `aws_eks`, `aws_vpc`, or `aws_cloudtrail`. XCOR determines the
  service from the log group name, log stream name, log format, or a stream parameter.
  Logs it can't attribute to an AWS service get `aws_other` and are stored like any
  other log.

Keep the default AWS log group names. The exception is Amazon ECS: XCOR
recognizes ECS logs only in log groups under `/aws/ecs/` or `/aws/vendedlogs/ecs/`, so
the `/ecs/<task definition family>` group that the ECS console creates gets
`aws_other`. XCOR can still recognize some services from custom log group names
by using the log stream name or format. Other logs get `aws_other`.

Every log also carries these fields, set from the delivery rather than the log
message:

| Field | Value |
| :- | :- |
| `aws.account_id` | The account that owns the log group, or the Firehose stream's account for direct delivery |
| `aws.region` | The log group's region, when available, or the Firehose stream's region |
| `aws.firehose.arn`, `aws.firehose.stream`, `aws.firehose.request_id` | The stream and request |
| `aws.firehose.attributes.<name>` | Each parameter you set on the stream |

Logs delivered through a CloudWatch Logs subscription also carry these fields:

| Field | Value |
| :- | :- |
| `aws.log_group`, `aws.log_stream` | Where CloudWatch Logs stored the log |
| `aws.subscription_filters`, `aws.event_id` | From the CloudWatch Logs subscription |
| `aws.timestamp` | When CloudWatch Logs stored the event |

For an organization trail, `aws.account_id` is the management account.
CloudTrail's `recipientAccountId` field identifies the account for each event, and
its `awsRegion` field identifies the event's region.

XCOR parses a JSON log message into fields and stores any other message in
`message`. For the following fields, values from the log message take precedence over
values that XCOR sets:

* Time: The log's `timestamp` field, then CloudTrail's `eventTime`, then the time
  CloudWatch Logs stored the event. A log sent straight to Firehose without a time
  gets the time XCOR receives it.
* Service: The `service` field is the `xcor_integration` value without `aws_`, such
  as `cloudtrail`, `eks`, or `rds`. Lambda logs get the function name. Logs labeled
  `aws_other` get the log group, or `aws_other` when there's no log group.
* Message: For CloudTrail, `message` is the event name, such as `AssumeRole`.
* Severity: CloudTrail events with an `errorCode` get `ERROR`, except
  `Client.DryRunOperation`, which means a dry run would have succeeded. Other
  CloudTrail events get `INFO`, and other logs without a severity get your
  organization's default.

## Troubleshooting

Firehose records delivery errors in the log group `/aws/kinesisfirehose/<STREAM_NAME>`.

* `401` errors, or a `content-type: application/json` error whose raw response starts
  with `401`: The token isn't an API token for a restricted service account with
  write-only permission in this organization. After a token change, Firehose can keep
  using the old token for up to 10 minutes.
* Secrets Manager errors: The secret must be JSON with the key `api_key`, and the
  stream role needs `secretsmanager:GetSecretValue` on the secret. If the secret uses
  a customer managed AWS Key Management Service key, the role also needs
  `kms:Decrypt` on the key and access through the key policy.
* `404` errors, or a `content-type: application/json` error whose raw response starts
  with `404`: The endpoint URL is incorrect. It must end in
  `/api/v1/data/logs/ingest/integrations/aws`.
* Logs labeled `aws_other`: The log group has a custom name or is the ECS console's
  `/ecs/` group, or the logs came straight from a service that XCOR couldn't
  identify from the stream name or log format. Keep the default AWS log group names.
  For a stream that carries only one service's logs, set the stream parameter
  `xcor_integration` to a recognized AWS integration ID. For example, use `aws_vpc` for
  a stream that carries only VPC Flow Logs in a custom format.
* Delivery succeeds but some logs are missing: XCOR drops individual logs
  larger than 1 MiB, more than 24 hours old, or more than 24 hours in the future
  without rejecting the Firehose request. CloudWatch Logs can deliver older events,
  such as after a long outage. XCOR also skips corrupt compressed records and
  records that make the request's decompressed contents exceed 256 MiB.
* Duplicate logs: After an error, Firehose sends the entire batch again, so some
  records can arrive twice.

Records that still fail after the retry window go to the S3 bucket. XCOR
doesn't read the bucket, so those records aren't ingested unless they're sent again
within 24 hours of their timestamps.

## Remove log delivery

Until you remove it, the account-level policy keeps sending every log group in the
region to the stream, and Firehose continues billing for it.

* For Terraform, run `terraform destroy`.

* For CloudFormation, note the failed-data bucket's name, and then delete the stack:

  ```shell theme={null}
  aws cloudformation describe-stack-resource --region <REGION> \
    --stack-name cortex-xcor-aws-logs --logical-resource-id FailedDataBucket \
    --query StackResourceDetail.PhysicalResourceId --output text
  aws cloudformation delete-stack --region <REGION> --stack-name cortex-xcor-aws-logs
  ```

  Starting with template version 1.1.0, deleting the stack keeps the failed-data
  bucket, so leftover records can't make the delete fail. Its records still expire,
  and you can delete it with `aws s3 rb s3://<BUCKET> --force` when you no longer need
  them, where `<BUCKET>` is the name the first command printed. If you passed
  `ApiToken`, deleting the stack also deletes its secret immediately, without a
  recovery window.

* For a StackSet, delete the stack instances from the organizational units and
  regions you deployed to:

  ```shell theme={null}
  OP_ID=$(aws cloudformation delete-stack-instances --region <ADMIN_REGION> \
    --stack-set-name cortex-xcor-aws-logs \
    --deployment-targets OrganizationalUnitIds=<OU_ID> \
    --regions <REGION> <REGION> \
    --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=100,FailureTolerancePercentage=100 \
    --no-retain-stacks \
    --query OperationId --output text)
  ```

  Wait for the operation with the commands in step 5 of
  [Set up many accounts and regions](#set-up-many-accounts-and-regions), and then delete
  the StackSet. Each account keeps its failed-data bucket.

  ```shell theme={null}
  aws cloudformation delete-stack-set --region <ADMIN_REGION> --stack-set-name cortex-xcor-aws-logs
  ```

* In the console, delete the account-level policy, then the Firehose stream,
  subscription role, failed-data bucket, and the stream's error log group, which the
  console creates:

  ```shell theme={null}
  aws logs delete-account-policy --region <REGION> \
    --policy-name cortex-xcor-logs --policy-type SUBSCRIPTION_FILTER_POLICY
  aws logs delete-log-group --region <REGION> \
    --log-group-name /aws/kinesisfirehose/cortex-xcor-logs
  ```

* For individual subscriptions, remove each subscription with:

  ```shell theme={null}
  aws logs delete-subscription-filter --region <REGION> \
    --log-group-name <LOG_GROUP> --filter-name cortex-xcor-logs
  ```

If you use Terraform, delete resources manually, or delete a stack created from a
template earlier than version 1.1.0, empty the failed-data bucket first. Otherwise,
the bucket deletion fails. If you created the token secret yourself and passed
`ApiTokenSecretArn`, delete it too, because the setup doesn't own it. By default,
Secrets Manager schedules deletion with a 30-day recovery window. During that window,
another setup attempt with the same secret name fails. The
`--force-delete-without-recovery` option skips the recovery window. Deletion is
asynchronous, so wait and retry if the name remains unavailable:

```shell theme={null}
aws secretsmanager delete-secret --region <REGION> --secret-id cortex-xcor/aws-logs-api-token \
  --force-delete-without-recovery
```


## Related topics

- [Ingest AWS CloudWatch metrics](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/server/aws/metrics.md)
- [Ingest log data](https://docs-xcor.paloaltonetworks.com/ingest/logs.md)
- [Ingest Google Cloud logs](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/server/gcp/logs.md)
- [UpdateLogIngestConfig](https://docs-xcor.paloaltonetworks.com/tooling/api-info/definition/operations/UpdateLogIngestConfig.md)
- [CreateLogIngestConfig](https://docs-xcor.paloaltonetworks.com/tooling/api-info/definition/operations/CreateLogIngestConfig.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.