> ## Documentation Index
> Fetch the complete documentation index at: https://docs-xcor.paloaltonetworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Concourse CI

> Concourse build, job scheduling, resource check, worker, garbage collection, database and API metrics, scraped from web nodes' Prometheus endpoint.

The Concourse CI integration requires CXDOT Collector 1.13.0 or greater.

[Concourse](https://concourse-ci.org/) is an open source continuous integration system
that runs pipelines of resources and jobs on a pool of workers. Use the Concourse CI
integration to collect build, job scheduling, resource checking, worker, garbage
collection, database, and API metrics from the Concourse web nodes running in your
environment.

The Concourse CI integration supports Concourse 7.11 or greater.

## Supported telemetry types

The Concourse CI integration supports these telemetry types:

| Type | Supported |
| - | - |
| Logs | No |
| Metrics | Yes |
| Traces | No |
| Events | No |

## Choose a collection method

The CXDOT Collector collects Concourse metrics in one of two ways:

* Recommended: Turn on the Prometheus emitter on each web node and scrape its endpoint
  with the Concourse CI integration. You get the Concourse CI integration's metrics,
  its overview dashboard, and a health metric for each web node. The Prerequisites and
  Configure sections cover this method.
* Alternative: Keep the Datadog emitter and send its DogStatsD metrics to the Datadog
  integration. Choose this method if you can't switch a web node to the Prometheus
  emitter yet. You get the metric names that the emitter sends, and none of the
  Concourse CI integration's metrics, dashboard, or health metric. The Send DogStatsD
  metrics to the Datadog integration section covers this method.

## Prerequisites

The Concourse CI integration has the following prerequisites:

* Turn on the Prometheus metrics emitter on each Concourse web node by setting
  `CONCOURSE_PROMETHEUS_BIND_IP` and `CONCOURSE_PROMETHEUS_BIND_PORT`, or the
  `--prometheus-bind-ip` and `--prometheus-bind-port` flags of `concourse web`. With
  the Concourse Helm chart, set `concourse.web.prometheus.enabled` to `true`, which
  binds port `9391` and creates a Service for it. For more information, see
  [Metrics](https://concourse-ci.org/docs/operation/metrics/) in the Concourse
  documentation.

  A web node runs one metrics emitter, and a web node with more than one emitter
  configured fails to start with the error `multiple emitters configured`. To turn
  on the Prometheus emitter on a web node that sends its metrics through another
  emitter, remove that emitter's settings. The Prometheus emitter serves an endpoint
  that any number of readers can request, so move whatever reads the node's metrics
  today to the Prometheus endpoint when you switch.

  To confirm that a web node serves metrics, request `/metrics` on the Prometheus
  port from a machine that can reach it and confirm that the request returns a
  successful status code:

  ```shell theme={null}
  curl -i http://concourse-web.example.com:9391/metrics
  ```

* Make the Prometheus port reachable on each web node from the CXDOT Collector.

## Configure

The Concourse CI integration is enabled by default, but it collects nothing until you
configure targets. Add a CXDOT integration annotation to your Concourse web pods or to a
Service in front of them, or list the web nodes to monitor.

To configure the Concourse CI integration, follow these steps:

1. To collect from Concourse web pods in Kubernetes, add a CXDOT integration annotation
   to the pod template. The annotation key must end with the container name. For
   example, add the following annotation to your Concourse web workload:

   ```yaml theme={null}
   spec:
     template:
       metadata:
         annotations:
           integrations.cxdot.paloaltonetworks.com/concourse-web: |
             schema_version: v1.16
             concourse_ci:
               endpoints:
                 - endpoint: ${target.host}:${target.port.prometheus}
       spec:
         containers:
           - name: concourse-web
             image: concourse/concourse
             ports:
               - name: prometheus
                 containerPort: 9391
   ```

   The Collector replaces `${target.host}` with the pod IP address and
   `${target.port.prometheus}` with the container port named `prometheus`. Metrics
   from each annotated pod include that pod's Kubernetes metadata.

   With the Concourse Helm chart, add the annotation under `web.annotations`. The
   chart names the web container `RELEASE_NAME-web` and its Prometheus port
   `prometheus`, so end the annotation key with the container name that your release
   produces.

2. Alternatively, to collect from web nodes outside your Kubernetes cluster or at
   addresses that you choose, list each web node in `endpoints`. For example, add the
   following to the Helm values file for your CXDOT Collector:

   ```yaml theme={null}
   collection:
     cxdot:
       integrations:
         concourse_ci:
           endpoints:
             - endpoint: concourse-web-1.example.com:9391
             - endpoint: concourse-web-2.example.com:9391
             - endpoint: http://concourse-web-3.example.com:9391/metrics
   ```

   When you list endpoints, the integration collects from exactly those endpoints
   instead of from annotated pods and Services. In Kubernetes, the elected
   cluster Collector collects from each endpoint that you list, one time rather than
   once per node. For more information, see
   [architecture](https://docs-xcor.paloaltonetworks.com/ingest/xcor/collector/install/kubernetes/architecture.md).

   List an address that reaches one web node. Each request to an address that
   balances across replicas, such as a Kubernetes Service in front of several web
   pods, reaches a different replica, so the metrics mix the replicas' values. With
   one web node, the Service that the Concourse Helm chart creates for the Prometheus
   port reaches that node. For a Concourse deployment with several web nodes, list
   each node's own address. Metrics from listed endpoints don't carry Kubernetes
   metadata from the Concourse pods. Use the `server.address` and `server.port`
   attributes to identify each web node.

3. Optional: To collect from both annotated pods and listed endpoints, list the
   endpoints in a second integration instance. For example, add the following to the
   Helm values file for your CXDOT Collector:

   ```yaml theme={null}
   collection:
     cxdot:
       integrations:
         concourse_ci: {}
         concourse_ci/edge:
           endpoints:
             - endpoint: concourse-web-1.example.com:9391
   ```

   The `concourse_ci/edge` key defines a named instance for the listed endpoints. The
   bare `concourse_ci` key lists no endpoints, so the integration continues to collect
   from annotated pods and Services.

4. Optional: If you can't change the pod template, configure the Service in front of
   the Prometheus port with a CXDOT integration annotation instead. The
   `integrations.cxdot.paloaltonetworks.com/endpoints` key makes each ready web pod
   behind the Service a separate target, so the metrics don't mix the replicas' values.
   For example, add the following annotation to the Prometheus Service of your
   Concourse deployment:

   ```yaml theme={null}
   metadata:
     annotations:
       integrations.cxdot.paloaltonetworks.com/endpoints: |
         schema_version: v1.16
         concourse_ci:
           endpoints:
             - endpoint: ${target.host}:${target.port.prometheus}
   spec:
     ports:
       - name: prometheus
         port: 9391
         targetPort: 9391
   ```

   The Collector replaces `${target.host}` with the IP address of each web pod and
   `${target.port.prometheus}` with the pod port that the Service port named
   `prometheus` forwards to. The Collector reads these annotations only when
   `endpoints` is empty.

5. Optional: If the OpenMetrics integration also scrapes your Concourse web pods or
   Service, disable OpenMetrics on that workload to avoid duplicate metrics. For
   example, set `openmetrics: {enabled: false}` in a CXDOT integration annotation on the
   pod or Service. For more information, see
   [autodiscovery](https://docs-xcor.paloaltonetworks.com/ingest/xcor/collector/discover/kubernetes.md).

### Validate

To validate the Concourse CI integration, follow these steps:

1. In the [Live Telemetry Analyzer](https://docs-xcor.paloaltonetworks.com/investigate/analyze/telemetry-analyzer.md),
   filter for `__name__=cxdot.integration.target.health` and
   `cxdot.integration.name=concourse_ci`. Confirm that each web node reports `1`,
   identified by its `server.address` and `server.port` attributes.

2. In [Metrics Explorer](https://docs-xcor.paloaltonetworks.com/investigate/querying/metrics/explorer.md),
   run the following query after Concourse finishes builds:

   ```text theme={null}
   sum by ("server.address", "concourse.build.status") (rate({"concourse.build.finished"}[$__rate_interval]))
   ```

   Confirm that the query returns a time series for each web node and build status.

## Known limitations

* Concourse emits `concourse.api.concurrent_request.count` and
  `concourse.api.concurrent_request.limit_hits` only for API actions with a configured
  concurrent request limit.
* Concourse emits `concourse.log.errors` only when error metric capture is enabled.

## Send DogStatsD metrics to the Datadog integration

Set `CONCOURSE_DATADOG_AGENT_HOST` and `CONCOURSE_DATADOG_AGENT_PORT`, or the
`--datadog-agent-host` and `--datadog-agent-port` flags of `concourse web`, to the
DogStatsD listener of the
[Datadog](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector/datadog.md)
integration. The listener accepts UDP on port `9125` by default.

The Concourse CI integration doesn't collect these DogStatsD metrics. The emitter sends
each metric as a gauge named with the `--datadog-prefix` value followed by the event
name, such as `concourse.ci.builds_running` with the Concourse Helm chart's default
prefix. The metric names on this page and its dashboard don't apply to those metrics.

## Configuration reference

Configure one Concourse CI integration instance with the following settings. In Helm values,
place these settings under `collection.cxdot.integrations.concourse_ci`. In a Collector
configuration file, place them under `cxdot.integrations.concourse_ci`.

### Optional settings

* **`enabled`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to enable this Concourse CI integration instance. If true, the Collector collects
  metrics from the web nodes that this instance is configured for. If false, the Collector
  doesn't run this integration instance.

* **`endpoints`**
  Type: `array of object`. Optional. Default: `[]`.
  Concourse web nodes to monitor, by the address of each node's Prometheus metrics endpoint. The
  integration instance collects only from the endpoints listed here. Specify `host:port` to
  request the `/metrics` path over HTTP, or a full HTTP or HTTPS URL. For a URL without a path,
  the Collector requests `/metrics`. A web node serves the endpoint only when its Prometheus
  emitter is on, set with `CONCOURSE_PROMETHEUS_BIND_IP` and `CONCOURSE_PROMETHEUS_BIND_PORT`.
  The Concourse Helm chart uses port `9391`. Each entry is one target, so list an address that
  reaches one web node. An address that balances requests across replicas, such as a Kubernetes
  Service with several pods, reaches a different replica on each collection.

* **`endpoints[].endpoint`**
  Type: `string`. Required.
  Address of one Concourse web node's Prometheus metrics endpoint, in `host:port` format or as a
  full HTTP or HTTPS URL. For `host:port`, the Collector requests the `/metrics` path over HTTP.
  A full URL is requested as given, such as `http://concourse-web.example.com:9391/metrics`. For
  a URL without a path, the Collector requests `/metrics`.

* **`collection_interval`**
  Type: `duration`. Optional. Default: `15s`.
  How often the integration collects metrics from each Concourse web node.

* **`timeout`**
  Type: `duration`. Optional. Default: `10s`.
  Maximum time the Collector waits for one collection from a web node. The value must be greater
  than zero. A value longer than `collection_interval` is lowered to `collection_interval`.


## Related topics

- [CXDOT Collector integrations](https://docs-xcor.paloaltonetworks.com/ingest/xcor/integrations/collector.md)
- [Service accounts](https://docs-xcor.paloaltonetworks.com/administer/accounts-teams/service-accounts.md)
- [Palo Alto Networks Telemetry Pipeline release notes](https://docs-xcor.paloaltonetworks.com/ingest/pipeline/v2/release-notes.md)
- [Convert Datadog monitors](https://docs-xcor.paloaltonetworks.com/ingest/metrics-traces/collector/mappings/datadog/convert-monitors.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.