What it collects
- CloudWatch Logs: Logs that AWS services and your applications write to CloudWatch log groups. Use an account-level subscription policy to cover every log group in a region, including new ones, or add a subscription filter to each log group. Subscriptions cover only log groups in the Standard log class. CloudTrail can also send its events to a log group.
- Logs sent straight to Firehose: Services such as Virtual Private Cloud (VPC) Flow Logs and Web Application Firewall (WAF) can deliver logs to a Firehose stream without a log group.
- Not collected: Logs that AWS writes only to Amazon S3, such as Classic Load Balancer access logs.
How it works
The following diagram shows how AWS log sources reach XCOR and where Firehose stores eligible failed deliveries.Before you begin
- AWS permissions: The identity that runs the setup needs the permissions listed in Required permissions.
- An XCOR API token: Create or use an API token for a restricted service account with write-only permission. For more information, see Create a restricted service account.
- Your organization name: The
<ORG_NAME>segment inhttps://<ORG_NAME>.chronosphere.io.
-
Existing subscriptions: An account can have one account-level subscription policy
in each region, and each log group allows five subscription filters of its own,
which the account-level policy doesn’t count against. If another tool already
uses the account-level policy, subscribe log groups individually. To inspect the
account policies, run:
Required permissions
To run the setup using the AWS provider for Terraform or CloudFormation in one account, the identity that runs it needs the following permissions. These permissions allow the identity to create and remove the setup resources. StackSets and the console need additional permissions.- Amazon Data Firehose
- Actions:
CreateDeliveryStream,DeleteDeliveryStream,DescribeDeliveryStream,UpdateDestination,ListTagsForDeliveryStream,TagDeliveryStream,UntagDeliveryStream - Resource: The stream
- Actions:
- Amazon S3
- Actions:
CreateBucket,DeleteBucket,PutLifecycleConfiguration,PutBucketTagging,ListBucket,Get*, andDeleteObjectto empty the bucket when you remove it - Resources: The failed-data bucket and its objects
- Actions:
- CloudWatch Logs
- Actions:
CreateLogGroup,DeleteLogGroup,PutRetentionPolicy,CreateLogStream,DeleteLogStream,DescribeLogGroups,DescribeLogStreams,ListTagsForResource,TagResource,PutAccountPolicy,DeleteAccountPolicy,DescribeAccountPolicies,PutSubscriptionFilter,DeleteSubscriptionFilter,DescribeSubscriptionFilters - Resource: All resources, because account policies have no resource ARN
- Actions:
- IAM
- Actions:
CreateRole,DeleteRole,GetRole,PutRolePolicy,GetRolePolicy,DeleteRolePolicy,ListRolePolicies,ListAttachedRolePolicies,ListInstanceProfilesForRole,PassRole - Resources: The two roles the setup creates
- Actions:
- AWS Secrets Manager
- Actions:
CreateSecret,DeleteSecret,DescribeSecret,UpdateSecret,GetSecretValue,TagResource,UntagResource - Resource: The token secret
- Actions:
- CloudFormation, for the CloudFormation tab only
- Actions:
CreateChangeSet,DescribeChangeSet,ExecuteChangeSet,DeleteChangeSet,DescribeStacks,DescribeStackEvents,DeleteStack,GetTemplateSummary - Resource: The stack
- Actions:
- Firehose role: Trusted by Firehose in your account. It can write to the failed-data bucket, read the token secret, and write the stream’s error log.
- Subscription role: Trusted by CloudWatch Logs in your account. It can put records into the stream.
kms:Decrypt permission on the key to the Firehose role, and grant the role access
in the key policy.
The setup has no access to your existing buckets or other resources, and it doesn’t
send logs written before it’s in place.
Firehose settings
Use these settings regardless of how you create the stream: Firehose bills each CloudWatch Logs record at its compressed size, rounded up to 5 KB. Log groups that send few events per batch cost more per log. Additional AWS data transfer charges can apply. Firehose sends a failed batch again on each retry, so data transfer can grow while deliveries fail.- Source:
Direct PUT. - Destination:
HTTP Endpoint. - HTTP endpoint URL:
https://<ORG_NAME>.chronosphere.io/api/v1/data/logs/ingest/integrations/aws. - Authentication: An AWS Secrets Manager secret containing
{"api_key":"<API_TOKEN>"}. Firehose picks up a rotated token within about 10 minutes. Using the API token as the endpoint access key also works, but stores it in the stream settings. - Content encoding:
GZIPfor smaller requests and lower data transfer costs. - Retry duration:
7200seconds, the longest Firehose allows. Records that still fail go to the S3 bucket. - Buffer size:
1MiB. Keep this value at 4 MiB or less. Firehose treats the size as a hint and encodes each record in a text representation larger than the source record. XCOR rejects requests over 16 MiB with a413, and Firehose drops a batch that gets a413without retrying it or backing it up. - Buffer interval:
60seconds. - Decompression and message extraction: Off. XCOR decodes CloudWatch Logs records. Message extraction removes the log group and log stream used to identify the service.
- Source record backup in Amazon S3:
Failed data only. - Backup bucket lifecycle: Expire objects after
14days. XCOR drops retried logs more than 24 hours old. - Parameters: None. For the exception for VPC Flow Logs in a custom format, see Send logs straight to Firehose.
Set up log delivery
Set up delivery in each account and region you want logs from. The Integrations Hub has no configuration form for AWS logs. Configure delivery in AWS by using one of the following methods.- Terraform
- AWS Management Console
- CloudFormation
-
Store your API token in AWS Secrets Manager in the region you’re setting up. Note
the secret’s ARN.
Read the token without echoing it, so it stays out of your shell history. Run the
readcommand on its own:Then create the secret: -
The configuration was tested with version 6 of the
awsprovider. The configuration also requires thetimeprovider. Set the region on theawsprovider: -
Set values for these variables, and apply the configuration in each account and
region. It creates the Firehose stream, S3 bucket for failed data with a lifecycle
rule, and IAM roles. Unless you turn it off, it also creates an account-level
subscription policy that sends every log group in the region except the stream’s
own error logs.
Terraform example
-
If you set
create_account_policy = false, follow Subscribe individual log groups with thestream_arnandsubscription_role_arnoutputs.
Set up many accounts and regions
Set up one stream in each account and region you want logs from. Each stream reports its own deliveries, so a problem appears in the account and region it affects. To deploy to many accounts in an organization in AWS Organizations, run the CloudFormation template as a StackSet with service-managed permissions. Pass the API token asApiToken instead of ApiTokenSecretArn. Each stack instance then creates
its own secret, so accounts added later through automatic deployment need no extra setup,
unless they already have an account-level subscription policy. CloudFormation shows
ApiToken as ****, but keeps it as a StackSet parameter. If the token should live
only in Secrets Manager, set up each account with ApiTokenSecretArn instead.
A StackSet exists only in the region where you create it. Choose one region to
administer it from, and pass that region as --region <ADMIN_REGION> on every
StackSet command, including later rotations and removal. With a delegated
administrator, pick one of the
regions AWS supports for delegated administrators.
Before you start,
enable all features
in AWS Organizations and
activate trusted access
for StackSets from the organization’s management account:
--call-as DELEGATED_ADMIN to every StackSet
command on this page: create-stack-set, create-stack-instances,
update-stack-instances, update-stack-set, describe-stack-set-operation,
list-stack-instances, delete-stack-instances, and delete-stack-set. The
delegated administrator also needs organizations:ListDelegatedAdministrators.
The following commands read the API token with read -rs. This command works in bash
and zsh and doesn’t show the token as you paste it. Run the read command on its
own, before the commands that use the token.
-
Read the API token:
-
Write the parameters to a file only you can read, so the token stays out of your
shell history:
-
Create the StackSet from the template in the CloudFormation tab, saved as
cortex-xcor-aws-logs.yaml, and then delete the parameters file:With automatic deployment, an account that joins a target organizational unit gets a stack, and an account that leaves has its stack deleted. The deleted stack’s failed-data bucket is kept. -
Deploy the StackSet to your organizational units and regions.
<OU_ID>can be the organization’s root ID (r-...) to cover every account.These preferences deploy to every account and region at once and keep going past accounts that fail. Without them, CloudFormation deploys to one account at a time and stops at the first failure, which also skips every later region. The preferences don’t apply to automatic deployments. -
Wait for the operation to finish, and then list the stack instances that failed.
With these preferences, the operation reports
SUCCEEDEDeven when some instances failed.
CreateAccountPolicy to false, wait for the operation with the
commands in step 5, and then subscribe log groups individually in that account with
the DeliveryStreamArn and SubscriptionRoleArn outputs of its stack:
ApiToken parameter, keeping the other parameters:
Subscribe individual log groups
To send only some log groups, or when another tool holds the account-level policy, add a subscription filter to each log group:Send logs straight to Firehose
Some services deliver logs to a Firehose stream without a log group. Give each service its own stream, created in the console with the same settings under a unique name. By default, both the Terraform configuration, which uses the AWS provider, and the CloudFormation configuration create the account subscription policy, so don’t run them again for these streams. The account-level policy sends each additional stream’s error log group,/aws/kinesisfirehose/<STREAM_NAME>, to the
stream targeted by the policy unless you add the error log group to the policy
exclusion.
XCOR recognizes these logs on its own:
- AWS WAF: By the stream name, which AWS requires to start with
aws-waf-logs-, and by the log format. - Amazon API Gateway: REST API access logs, identified by a stream name that starts
with
amazon-apigateway-. - VPC Flow Logs: By the default log format. For a custom format, add the stream
parameter
xcor_integrationwith the valueaws_vpc.
xcor_integration parameter to a
recognized AWS integration ID, such as aws_cloudtrail, aws_eks, aws_lambda, or
aws_rds. XCOR ignores unknown values. Logs it can’t otherwise recognize are
stored with the label aws_other.
Validate log ingestion
After setup, allow at least one minute for the configured Firehose buffer interval before validating ingestion.- In the Amazon Data Firehose console, open the stream. HTTP endpoint delivery success should report 100%, and Records delivered to HTTP endpoint should be non-zero. A successful delivery means XCOR accepted the request. It doesn’t confirm that every log in the request passed ingestion validation.
-
In the navigation menu, select
Explorers > Logs Explorer
and run:
Each AWS service sending logs appears as a row. For more about queries, see Search and filter log data.
Log labels and fields
Every log gets two integration labels:xcor.integration.name: alwaysaws, the integration’s catalog ID.xcor_integration: the AWS service that wrote the log, such asaws_lambda,aws_rds,aws_eks,aws_vpc, oraws_cloudtrail. XCOR determines the service from the log group name, log stream name, log format, or a stream parameter. Logs it can’t attribute to an AWS service getaws_otherand are stored like any other log.
/aws/ecs/ or /aws/vendedlogs/ecs/, so
the /ecs/<task definition family> group that the ECS console creates gets
aws_other. XCOR can still recognize some services from custom log group names
by using the log stream name or format. Other logs get aws_other.
Every log also carries these fields, set from the delivery rather than the log
message:
Logs delivered through a CloudWatch Logs subscription also carry these fields:
For an organization trail,
aws.account_id is the management account.
CloudTrail’s recipientAccountId field identifies the account for each event, and
its awsRegion field identifies the event’s region.
XCOR parses a JSON log message into fields and stores any other message in
message. For the following fields, values from the log message take precedence over
values that XCOR sets:
- Time: The log’s
timestampfield, then CloudTrail’seventTime, then the time CloudWatch Logs stored the event. A log sent straight to Firehose without a time gets the time XCOR receives it. - Service: The
servicefield is thexcor_integrationvalue withoutaws_, such ascloudtrail,eks, orrds. Lambda logs get the function name. Logs labeledaws_otherget the log group, oraws_otherwhen there’s no log group. - Message: For CloudTrail,
messageis the event name, such asAssumeRole. - Severity: CloudTrail events with an
errorCodegetERROR, exceptClient.DryRunOperation, which means a dry run would have succeeded. Other CloudTrail events getINFO, and other logs without a severity get your organization’s default.
Troubleshooting
Firehose records delivery errors in the log group/aws/kinesisfirehose/<STREAM_NAME>.
401errors, or acontent-type: application/jsonerror whose raw response starts with401: The token isn’t an API token for a restricted service account with write-only permission in this organization. After a token change, Firehose can keep using the old token for up to 10 minutes.- Secrets Manager errors: The secret must be JSON with the key
api_key, and the stream role needssecretsmanager:GetSecretValueon the secret. If the secret uses a customer managed AWS Key Management Service key, the role also needskms:Decrypton the key and access through the key policy. 404errors, or acontent-type: application/jsonerror whose raw response starts with404: The endpoint URL is incorrect. It must end in/api/v1/data/logs/ingest/integrations/aws.- Logs labeled
aws_other: The log group has a custom name or is the ECS console’s/ecs/group, or the logs came straight from a service that XCOR couldn’t identify from the stream name or log format. Keep the default AWS log group names. For a stream that carries only one service’s logs, set the stream parameterxcor_integrationto a recognized AWS integration ID. For example, useaws_vpcfor a stream that carries only VPC Flow Logs in a custom format. - Delivery succeeds but some logs are missing: XCOR drops individual logs larger than 1 MiB, more than 24 hours old, or more than 24 hours in the future without rejecting the Firehose request. CloudWatch Logs can deliver older events, such as after a long outage. XCOR also skips corrupt compressed records and records that make the request’s decompressed contents exceed 256 MiB.
- Duplicate logs: After an error, Firehose sends the entire batch again, so some records can arrive twice.
Remove log delivery
Until you remove it, the account-level policy keeps sending every log group in the region to the stream, and Firehose continues billing for it.-
For Terraform, run
terraform destroy. -
For CloudFormation, note the failed-data bucket’s name, and then delete the stack:
Starting with template version 1.1.0, deleting the stack keeps the failed-data bucket, so leftover records can’t make the delete fail. Its records still expire, and you can delete it with
aws s3 rb s3://<BUCKET> --forcewhen you no longer need them, where<BUCKET>is the name the first command printed. If you passedApiToken, deleting the stack also deletes its secret immediately, without a recovery window. -
For a StackSet, delete the stack instances from the organizational units and
regions you deployed to:
Wait for the operation with the commands in step 5 of Set up many accounts and regions, and then delete the StackSet. Each account keeps its failed-data bucket.
-
In the console, delete the account-level policy, then the Firehose stream,
subscription role, failed-data bucket, and the stream’s error log group, which the
console creates:
-
For individual subscriptions, remove each subscription with:
ApiTokenSecretArn, delete it too, because the setup doesn’t own it. By default,
Secrets Manager schedules deletion with a 30-day recovery window. During that window,
another setup attempt with the same secret name fails. The
--force-delete-without-recovery option skips the recovery window. Deletion is
asynchronous, so wait and retry if the name remains unavailable: