--- title: Prometheus Cardinality Explosion on AWS: AMP, EMF, and Cost-Aware Metrics description: That `user_id` label on every HTTP metric turns Amazon Managed Prometheus into a five-figure line item. This guide explains cardinality mechanics, EMF vs remote write, and Application Signals defaults worth disabling. url: https://www.joinhorizon.co/blog/prometheus-cardinality-explosion-amp-cloudwatch-cost-control/ datePublished: 2026-06-12T00:00:00.000Z dateModified: 2026-10-28T00:00:00.000Z author: manojkumar-varatharajan category: DevOps & CI/CD tags: engineering-guide, observability, prometheus, cloudwatch, aws, finops --- # Prometheus Cardinality Explosion on AWS: AMP, EMF, and Cost-Aware Metrics > That `user_id` label on every HTTP metric turns Amazon Managed Prometheus into a five-figure line item. This guide explains cardinality mechanics, EMF vs remote write, and Application Signals defaults worth disabling. **Amazon Managed Prometheus (AMP) pricing (June 2026)** scales with **metrics ingested and stored**, so cardinality is dollars. A single histogram with `path` label including UUIDs can create **millions of active series** within hours. Since **July 31, 2026**, there is a second Prometheus path on AWS: [CloudWatch managed collectors](https://www.joinhorizon.co/blog/cloudwatch-managed-prometheus-collectors-july-2026/) can scrape the same `/metrics` exporters into a **CloudWatch OpenTelemetry dataset** at **$0.50/GB ingested** (15 months storage, no per-series custom-metric fee) instead of AMP samples, **if** your pane is CloudWatch and you fix labels before scrape. Cardinality still multiplies OTLP GB. The billing meter changes, but the hygiene stays the same. For AWS Example 22 shape (100 hosts × 50 metrics @ 60s), OTLP is **$54/mo** vs **$1,500/mo** classic custom metrics. See the collectors post cost CSV. AMP remains the right backend when you need recording rules and Grafana-native workflows. > **Benchmark pattern:** OTel demo workload on EKS: enabling `http.route` with raw URL paths pushed active series from **12k → 890k** in 6 h; AMP estimate **+$2,400/mo**. Relabel to template routes (`/users/{id}`) restored **14k** series. See [observability beyond CloudWatch](https://www.joinhorizon.co/blog/aws-observability-beyond-cloudwatch-otel-prometheus-grafana-2026/) for stack wiring. ## Symptom → mechanism → AWS control | Production symptom | Mechanism | AWS control | | ---------------------------------- | -------------------------------------- | --------------------------------------------------- | | Observability bill exceeds compute | High-cardinality labels (user_id, pod) | AMP relabel_configs drop, aggregate recording rules | | CloudWatch PutMetricData throttled | Custom metric cardinality limits | EMF embedded metrics with bounded dimensions | | Dashboard slow to load | Million-series PromQL scan | AMP query logging, cardinality governance policy | **Opinionated take:** Treat metric labels like database indexes: approve new high-cardinality labels in review and drop pod-name from application metrics at scrape time. ## Mechanism Prometheus identifies a time series by **metric name + label set**. Each unique combination is billed storage and query cost. **High-cardinality labels** (IDs) multiply series combinatorially with other labels (`status`, `method`, `pod`). ## AWS controls | Approach | Service | Use when | | ---------------- | --------------------------------------------------------------------- | ------------------------------------------------- | | Managed scrape | CloudWatch managed collectors → OTLP (Jul 2026) | CloudWatch-first; metrics-only scrape fleet today | | Managed backend | AMP + AMG | EKS/ECS metrics at scale; Grafana/recording rules | | Embedded metrics | CloudWatch EMF | Lambda/custom apps without scrape | | SLO-native | Application Signals | Service golden signals; watch auto-discovered ops | | Cost guard | Metric filters + alarms on `IncomingLogEvents` / AMP workspace limits | FinOps gate | **Opinionated take:** **Relabel at scrape time** using managed collectors, ADOT, or Prometheus `metric_relabel_configs`, before remote_write or OTLP export. Do not fix cardinality in Grafana dashboards. For destination choice (CloudWatch OTLP vs AMP vs classic custom metrics), see [CloudWatch managed Prometheus collectors (July 2026)](https://www.joinhorizon.co/blog/cloudwatch-managed-prometheus-collectors-july-2026/). ## AWS services map | Need | Service | Skip when | | ------------------- | ----------------------------------------- | --------------------------------------- | | Cardinality control | AMP + relabel/aggregate rules | CloudWatch only with <10 custom metrics | | Cost attribution | Cost and Usage Report + AMP usage metrics | Single-service app with default metrics | | FinOps guardrails | CloudWatch anomaly detection on AMP spend | Pre-production with negligible ingest | ## When this advice breaks - **Short-lived batch jobs:** High churn series may be acceptable if retention is 24h and jobs are few. - **Debugging incidents:** Temporary high-cardinality scrape OK with documented TTL and owner. ## What to do this week 1. Export top 20 labels by series count from AMP or Prometheus `label_values` sampling. 2. Add `drop`/`labelmap` processors in ADOT config for forbidden labels (`user_id`, `trace_id`). 3. Set CloudWatch alarm on AMP `DiscardedSamples`, OTel ingestion GB spike, or workspace ingestion rate spike. 4. If CloudWatch is your pane and you scrape `/metrics` today, compare AMP samples vs [managed collector → OTLP](https://www.joinhorizon.co/blog/cloudwatch-managed-prometheus-collectors-july-2026/) with the published cost CSV before the next scrape fleet expansion. 5. Pair with log sampling guide (part 3 of this track). ## More in This Track Part of the **Engineering Guides** library (June 2026). - Previous: [Part 1](https://www.joinhorizon.co/blog/aws-observability-beyond-cloudwatch-otel-prometheus-grafana-2026/) - Next: [Part 3](https://www.joinhorizon.co/blog/log-aggregation-sampling-cloudwatch-otel-aws/) - Browse tracks: [Engineering Guides hub](https://www.joinhorizon.co/resources/engineering-guides/) ## What this guide doesn't cover Distributed tracing propagation is covered in part 1, the OTel guide in this track. ## FAQ ### What is a safe cardinality budget per metric? Keep high-cardinality dimensions (user ID, request ID, URL path with IDs) off Prometheus labels. As a rule: if a label can take more than ~100 unique values in production, it belongs in logs or traces, not metrics. ### Should I use AMP or self-managed Prometheus on EKS? AMP removes ops toil and integrates with managed Grafana; cost is ingestion + storage driven by active series. Self-managed saves service fees but not cardinality mistakes. You still pay EC2 and ops time. --- *Source: https://www.joinhorizon.co/blog/prometheus-cardinality-explosion-amp-cloudwatch-cost-control/*