---
title: Log Aggregation and Intelligent Sampling with CloudWatch and OpenTelemetry
description: Ingesting every debug log to CloudWatch is how observability becomes a FinOps incident. Tail sampling with ADOT, Logs Insights, and Firehose to S3 for the long tail.
url: https://www.joinhorizon.co/blog/log-aggregation-sampling-cloudwatch-otel-aws/
datePublished: 2026-06-12T00:00:00.000Z
dateModified: 2026-10-28T00:00:00.000Z
author: manojkumar-varatharajan
category: DevOps & CI/CD
tags: engineering-guide, observability, cloudwatch, opentelemetry, aws
---

# Log Aggregation and Intelligent Sampling with CloudWatch and OpenTelemetry

> Ingesting every debug log to CloudWatch is how observability becomes a FinOps incident. Tail sampling with ADOT, Logs Insights, and Firehose to S3 for the long tail.

**CloudWatch Logs ingestion (June 2026)** bills per GB. **100% trace/log correlation** without sampling destroyed margins on a $40k/mo observability line item for a mid-market SaaS we benchmarked.

## Symptom → mechanism → AWS control

| Production symptom            | Mechanism                    | AWS control                                                         |
| ----------------------------- | ---------------------------- | ------------------------------------------------------------------- |
| Log bill dwarfs compute       | 100% log ingest at INFO      | OTel probabilistic + tail sampling, CloudWatch Logs retention tiers |
| Can't find error in log flood | No trace correlation         | OTel trace_id in log attributes, CloudWatch Logs Insights           |
| Hot partition on log group    | Single log group per service | Per-environment log groups, S3 export for archive                   |

**Opinionated take:** Sample success logs at 1 to 5% and keep 100% of errors. Wire trace_id into every log line before you centralize aggregation.

> **Benchmark pattern (hypothetical workload):** 200GB/day application logs, OTel tail-sampling (1% success, 100% error) reduces ingest to 22GB/day, CloudWatch Logs bill $1,840→$202/month; X-Ray trace-linked logs preserve full context on errors.

## Aggregation architecture

1. **App** → structured JSON (correlation ID)
2. **ADOT collector** → tail sampling (keep errors + slow)
3. **CloudWatch Logs** hot path + **Firehose** → S3/Glue for audit

## Sampling rules

- Always keep: `level=ERROR`, `http.status>=500`, latency > SLO
- Sample info: 1 to 5% baseline
- Never sample security audit events

## Logs Insights

Use for incident search, not as a primary metrics store. Pair with [cardinality guide](https://www.joinhorizon.co/blog/prometheus-cardinality-explosion-amp-cloudwatch-cost-control/).

## AWS services map

| Need                  | Service                            | Skip when                                |
| --------------------- | ---------------------------------- | ---------------------------------------- |
| Intelligent sampling  | ADOT collector tail_sampling       | Compliance requires 100% audit retention |
| Log storage + query   | CloudWatch Logs + Insights         | Long-term archive → S3 + Athena          |
| Trace-log correlation | OTel + X-Ray / Application Signals | Batch jobs with no request context       |

## What to do this week

1. Enable ADOT tail sampling processor in collector config.
2. Set log retention tiers (7d hot, 90d S3).
3. Dashboard ingestion GB/day with anomaly detection.

## More in This Track

Part of the **Engineering Guides** library (June 2026).

- Previous: [Part 2](https://www.joinhorizon.co/blog/prometheus-cardinality-explosion-amp-cloudwatch-cost-control/)
- Next: [Part 4](https://www.joinhorizon.co/blog/aws-resilience-retries-circuits-graceful-shutdown/)
- Browse tracks: [Engineering Guides hub](https://www.joinhorizon.co/resources/engineering-guides/)

## What this guide doesn't cover

Full OTel stack setup is covered in part 1 of this track.

## FAQ

### When should I not sample CloudWatch logs?
Never sample security audit events. Keep 100% of ERROR, HTTP 5xx, and latency-over-SLO lines. Sample info-level success at 1 to 5%. Compliance that requires full retention belongs in S3 via Firehose, not an unsampled hot log group.

### Will Logs Insights replace metrics?
No. Insights is for incident search. Cardinality and SLO math belong in metrics; see the AMP cardinality guide linked from this post. Centralizing every debug line is how observability becomes a FinOps incident.

### What could go wrong if I ingest 100% of INFO logs?
The bill dwarfs compute. A mid-market SaaS pattern we benchmarked hit about $40k/mo on unsampled ingest. Tail sampling with ADOT (keep errors and slow traces) is the control, not “turn off logging.”

### How do I still find an error after sampling?
Put trace_id on every structured JSON line and keep 100% of errors. OTel tail sampling plus X-Ray or Application Signals preserves full context on the failure path.

---

*Source: https://www.joinhorizon.co/blog/log-aggregation-sampling-cloudwatch-otel-aws/*