---
title: AWS Observability Costs: Cardinality Budgets & FinOps Limits
description: CloudWatch Logs Insights bills $0.005 per GB scanned and high-cardinality custom metrics multiply costs. Cardinality budgets, sampling rules, and FinOps fixes.
url: https://www.joinhorizon.co/blog/aws-observability-finops-cardinality-cost-control/
datePublished: 2026-05-08T00:00:00.000Z
dateModified: 2026-10-28T00:00:00.000Z
author: palaniappan-p
category: Cloud Architecture
tags: amazon-cloudwatch, observability, opentelemetry, aws-xray, finops, cost-optimization
---

# AWS Observability Costs: Cardinality Budgets & FinOps Limits

> CloudWatch Logs Insights bills $0.005 per GB scanned and high-cardinality custom metrics multiply costs. Cardinality budgets, sampling rules, and FinOps fixes.

On **May 8, 2026**, **Amazon CloudWatch** observability bills bite hardest where teams least audit them: **Logs Insights** at **$0.005 per GB scanned** (US East list pricing) and **custom metrics** that explode when a per-user-ID dimension lands on a hot path. A "cheap" wide query scheduled every minute, or a `PutMetricData` storm on a high-cardinality dimension, becomes a five-figure monthly line item before finance reads the dashboard.

This guide is the **FinOps view** of AWS observability: **cardinality budgets**, **sampling rules**, and **query hygiene** that keep CloudWatch and OpenTelemetry costs predictable. For the broader CloudWatch capability surface (metrics, logs, alarms, Application Signals), see [CloudWatch observability best practices](https://www.joinhorizon.co/blog/aws-cloudwatch-observability-metrics-logs-alarms-best-practices/). For full FinOps remediation across compute and observability, see our [AWS cloud cost optimization services](https://www.joinhorizon.co/services/aws-cloud-cost-optimization-services/).

> **Reproduce this:** Paste starter queries from [`examples/architecture-blog-2026/observability/logs-insights-queries.txt`](https://www.joinhorizon.co/examples/architecture-blog-2026/observability/logs-insights-queries.txt) into Logs Insights; replace log groups and trace IDs.

## The three cost surfaces: metrics, logs, traces

- **Custom metrics** bill **per dimension combination**. Adding `user_id` to a metric with 500k DAU creates 500k time series, which is a database, not a metric.
- **Logs Insights** bills **per GB scanned** ($0.005/GB US East, May 2026). A `SELECT *` across 30 log groups for "the last hour" is bigger than people think.
- **Traces** bill **per-span ingested** in **AWS X-Ray**; without sampling rules, instrument-everything day one is the most expensive learning experience your team will have.

## Cardinality budgets

A single mis-applied tag can silently triple your observability bill.

- Set a **budget** per service: max distinct dimension combinations per metric, max distinct attributes per trace span. Review weekly.
- Prefer **aggregated** business metrics with bounded dimensions (region, env, tier) over per-user-ID metrics. Use **traces** for per-user debugging, not metrics.
- Use **embedded metric format (EMF)** carefully. EMF generates metrics from log lines, which means a runaway log can mint runaway metrics.

> **What broke:** A "temporary" debugging query (`SELECT *` across 30 log groups) landed in a CI cron for weeks. Finance flagged an **$18k** monthly delta. Logs were "cheap" until automation scaled humans out of the loop. Fix: query linting, required group filters, and budget alarms per log product.

## Sampling: the lever most teams skip

- **Head-based sampling** (sample at trace start) is simple and predictable; default to it before tail-based.
- **Tail-based sampling** requires a collector tier (**AWS Distro for OpenTelemetry**, OTel Collector). Useful for keeping all error traces while dropping normal ones, _if_ the rules do not accidentally keep 100% of "normal" anyway.
- **Validate trace counts against billing weekly.** Misconfigured rules look correct in code and silently re-include the world.

## OpenTelemetry adoption pacing

Half-instrumented OpenTelemetry can cost more than central **AWS X-Ray** with known sampling rules. Phase adoption: critical paths first, tail-based sampling second, broad rollout last. For chaos-driven OTel learning, see [OTel Demo Game](https://www.joinhorizon.co/blog/otel-demo-game-aws-observability-chaos-engineering/).

## Monitoring vs alerting and cost

**Monitoring** collects continuously; **alerting** routes to humans. The cost link: every alarm that fires more than twice a week without an action update is a candidate for deletion or re-tiering. Noisy alarms keep their underlying metric pipelines hot and humans paged.

> **Opinionated take:** If an alert fires more than twice a week without an action checklist update, delete or re-tier the alarm. Noisy paging is a security incident waiting to happen once fatigue sets in.

The [real cost of skipping 24/7 monitoring](https://www.joinhorizon.co/blog/real-cost-of-no-24-7-aws-monitoring/) quantifies the business risk of silent failure. Our [distributed debugging guide](https://www.joinhorizon.co/blog/debug-production-distributed-aws-systems/) gives the rule: **alert on metrics**, **triage with traces**, **diagnose with logs**.

## FinOps coupling

Observability spend belongs in the same review as compute. Filter Cost Explorer to **Amazon CloudWatch** (and related telemetry services), then use **Analyze with Amazon Q** (June 2026) to narrate which usage types drove the line item, which is faster than exporting CSVs for finance. Cross-references: [Cost Explorer guide](https://www.joinhorizon.co/blog/aws-cost-explorer-budgets-monitoring-guide/), [engineering cost ownership](https://www.joinhorizon.co/blog/aws-finops-gap-engineering-cost-ownership/), and [logging cost deep dive](https://www.joinhorizon.co/blog/aws-cloudwatch-logging-costs-observability/).

## What this post doesn't cover

- **Third-party APM** pricing negotiations: vendor-specific.
- **Security analytics** in Security Lake vs CloudWatch: different retention/compliance story ([Security Lake guide](https://www.joinhorizon.co/blog/amazon-security-lake-ocsf/) when OCSF normalization matters).
- **Capability deep dive** on metrics, logs, alarms, Application Signals lives in [CloudWatch observability best practices](https://www.joinhorizon.co/blog/aws-cloudwatch-observability-metrics-logs-alarms-best-practices/).

## If you only do one thing

Set a **per-service cardinality budget** and review it weekly with finance. The other cost levers follow from that one number.

## What to do this week

1. Inventory scheduled Logs Insights queries; delete or constrain time windows.
2. Audit every `PutMetricData` call for per-user-ID or per-request-ID dimensions; remove or move to traces.
3. Verify **W3C trace context** crosses service boundaries in staging (not just Lambda console traces).
4. Pair every new alarm with a **runbook link** or delete the alarm. No orphan pages.

When resilience patterns intersect telemetry, read [retries, circuits, and graceful shutdown](https://www.joinhorizon.co/blog/aws-resilience-retries-circuits-graceful-shutdown/).

## FAQ

### When should we avoid high-cardinality custom metrics in CloudWatch?
When dimensions explode (per-user IDs on hot paths) and billing scales nonlinearly. Prefer aggregated business metrics, sampling, or trace-derived insights for per-tenant debugging, not raw PutMetricData storms.

### What is the biggest observability cost trap on AWS?
Scheduled Logs Insights queries on wide time windows. List pricing is $0.005 per GB scanned (US East, May 2026); a "cheap" minute-by-minute query across many log groups becomes a five-figure monthly line item before finance flags it. Constrain time windows, narrow log groups, and require lint approval on scheduled queries.

### Is logging alone enough for production AWS systems?
Logs without correlation identifiers devolve into forensic archaeology across log groups. You need at least one of: structured trace IDs, correlated metrics, or traces. Our distributed debugging guide shows the composite workflow.

### When should we not adopt OpenTelemetry everywhere day one?
When teams lack collector discipline and cardinality budgets. Half-instrumented OTel can cost more than central X-Ray with known sampling rules. Phase adoption: critical paths first, tail-based sampling second.

### What breaks tail-based sampling budgets?
Misconfigured rules that still capture 100% of "normal" traffic while claiming tail sampling. Validate trace counts against billing weekly.

### How is this different from CloudWatch best practices coverage?
The companion CloudWatch observability best practices post covers the capability surface (metrics, logs, alarms, Application Signals). This post is the FinOps view: cardinality budgets, sampling rules, query hygiene, and the cost behaviors that make observability bills predictable.

---

*Source: https://www.joinhorizon.co/blog/aws-observability-finops-cardinality-cost-control/*