--- title: Observability Beyond CloudWatch (2026): When to Add Application Signals, ADOT, Managed Prometheus, and Grafana — and When Not To description: The reflex to bolt Amazon Managed Prometheus + Grafana onto every workload is how observability bills quietly double. CloudWatch Application Signals now gives you an auto-discovered service map, SLOs, and traces with near-zero setup; AMP only earns its keep when you are PromQL-native or drowning in high-cardinality metrics, where ingestion (not retention) is the cost driver. The post covers the decision matrix, an ADOT dual-export config, and the three levers that actually cut the AMP bill. url: https://www.joinhorizon.co/blog/aws-observability-beyond-cloudwatch-otel-prometheus-grafana-2026/ datePublished: 2026-06-09T00:00:00.000Z dateModified: 2026-10-28T00:00:00.000Z author: manojkumar-varatharajan category: DevOps & CI/CD tags: aws, observability, opentelemetry, devops, cost-optimization, engineering-guide --- # Observability Beyond CloudWatch (2026): When to Add Application Signals, ADOT, Managed Prometheus, and Grafana — and When Not To > The reflex to bolt Amazon Managed Prometheus + Grafana onto every workload is how observability bills quietly double. CloudWatch Application Signals now gives you an auto-discovered service map, SLOs, and traces with near-zero setup; AMP only earns its keep when you are PromQL-native or drowning in high-cardinality metrics, where ingestion (not retention) is the cost driver. The post covers the decision matrix, an ADOT dual-export config, and the three levers that actually cut the AMP bill. **The fastest way to double an AWS observability bill in 2026 is to bolt Amazon Managed Prometheus and Grafana onto a workload that CloudWatch Application Signals would have covered.** As of mid-2026, CloudWatch Application Signals gives you an auto-discovered service map, SLOs, and correlated traces with near-zero instrumentation, features that landed and matured across 2024 to 2025 (dependency SLOs in April 2025, multi-account views via OAM in February 2025, EKS auto-monitor in the CloudWatch Observability add-on v4.0.0 in May 2025). Yet the reflex is still to stand up a second metrics backend "for real observability." Sometimes that's right. Often it's a second billing surface and a second query language for no measured gain. This post is the decision framework. ## Symptoms, mechanisms, and AWS controls | Production symptom | Mechanism | AWS control | | ------------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------- | | Can't correlate traces to logs | Siloed CloudWatch log groups | ADOT collector with trace context propagation | | Missing service dependency map | No distributed tracing | X-Ray or OTel → CloudWatch Application Signals | | Prometheus ops burden | Self-managed scrape + storage | Managed collector → CloudWatch OTLP, AMP, or AMG | | Scrape fleet you only run for metrics | ADOT DaemonSet / Prometheus agent HA | [CloudWatch managed collectors](https://www.joinhorizon.co/blog/cloudwatch-managed-prometheus-collectors-july-2026/) (July 31, 2026) | **Opinionated take:** Stay on CloudWatch until you need cross-service RED metrics or PromQL, then pick a scrape destination deliberately. As of **July 31, 2026**, agentless [managed Prometheus collectors](https://www.joinhorizon.co/blog/cloudwatch-managed-prometheus-collectors-july-2026/) can replace a metrics-only ADOT scrape fleet when CloudWatch is the pane; reach for AMP when Grafana/recording rules are non-negotiable. Self-hosted Prometheus on EC2 is still the ops tax you should avoid. This is for platform and SRE teams, and the engineering leaders signing the observability invoice. We ship a [tier decision matrix](https://www.joinhorizon.co/examples/architecture-blog-2026/observability-stack/observability-tier-decision-matrix.md), an [ADOT dual-export collector config](https://www.joinhorizon.co/examples/architecture-blog-2026/observability-stack/adot-collector-config.yaml), the [three AMP cost-control levers](https://www.joinhorizon.co/examples/architecture-blog-2026/observability-stack/amp-cost-control.md), and an [AMG/AMP cost model CSV](https://www.joinhorizon.co/examples/architecture-blog-2026/observability-stack/observability-cost-model.csv). > **Benchmark pattern (not a cited client):** A composite Kubernetes-heavy platform: ~40 microservices on EKS, an existing Prometheus + Grafana habit, and a "scrape everything at 15s" default. Modeled in the [cost CSV](https://www.joinhorizon.co/examples/architecture-blog-2026/observability-stack/observability-cost-model.csv): moving non-alerting infra series from 15s to 60s cuts those samples ~75%, and adding source-side metric filtering takes the relative ingestion index from 100 to ~22, roughly a 4 to 5x reduction in the dominant AMP cost driver, with no loss of alerting fidelity. Separately, switching 8 dashboard-only engineers from Grafana Editor ($9) to Viewer ($5) trims AMG user cost by ~44% on that slice. Neither change touches retention, because storage isn't where the money is. ## Tier 1: CloudWatch core as the baseline If you need logs, metrics, alarms, and Logs Insights over AWS services, CloudWatch core is the answer, so don't add a second stack for it. The cost traps here are well-trodden: high-cardinality **custom metrics** and verbose log ingestion. Those are real, but they are CloudWatch _hygiene_ problems, not reasons to migrate to Prometheus. (We cover that hygiene in depth in [observability FinOps and cardinality cost control](https://www.joinhorizon.co/blog/aws-observability-finops-cardinality-cost-control/) and [CloudWatch logging costs](https://www.joinhorizon.co/blog/aws-cloudwatch-logging-costs-observability/).) ## Tier 2: Application Signals is the APM you probably already have ![Architecture diagram of an EKS workload sending OpenTelemetry through ADOT into CloudWatch Application Signals](../../assets/images/blog/aws-observability-beyond-cloudwatch-otel-prometheus-grafana-2026-signals.webp) _Figure: Application Signals in the article. ADOT, or the CloudWatch agent, sends telemetry into CloudWatch. Transaction Search is what unlocks the service map and traces. The article says to measure that before adding another backend. [Open the draw.io source](https://www.joinhorizon.co/examples/architecture-blog-2026/aws-observability-beyond-cloudwatch-otel-prometheus-grafana-2026/signals.drawio)._ The moment you want **APM** (a service map, SLOs, "which dependency is breaking my latency"), the default should be **CloudWatch Application Signals**, not a new backend. It auto-discovers services and dependencies, draws the application map, tracks period- and request-based SLOs (including SLOs _on dependencies_ since April 2025), and correlates traces so you can drill from a fault-rate summary to the offending span. It auto-instruments across EKS, EC2, ECS, Kubernetes, Lambda, and on-prem, and ingests OpenTelemetry via ADOT and the CloudWatch agent. **Setup gotcha:** you must **enable Transaction Search** to unlock the full APM feature set under the unified Application Signals pricing that bundles X-Ray traces and transaction spans. On EKS, the CloudWatch Observability add-on (v4.0.0+) can auto-monitor workloads behind a single config flag. **Service Events (July 6, 2026):** once Application Signals is enabled, AWS automatically captures exception/latency snapshots and deployment events for Java, Python, and JavaScript workloads (ADOT SDK or CloudWatch Observability EKS add-on), which is useful for answering "did this deploy introduce new errors?" without bolting on a second APM. Function-call metrics remain opt-in. **Opinionated take:** most teams reaching for "we need APM, let's deploy Grafana Tempo + a tracing backend" should enable Application Signals first and measure whether the gap is real. It usually isn't. ## Tier 3: Prometheus metrics and a single scrape destination On **July 31, 2026**, AWS added **CloudWatch managed Prometheus collectors**: agentless scrapers for EKS, EC2, ECS, MSK, and OpenSearch via the AMP `CreateScraper` API. Metrics can land in a **CloudWatch OpenTelemetry dataset** (PromQL + alarms in one pane) or an **AMP workspace**, but not both for the same job. That splits Tier 3 into four paths instead of one default stack: | Path | When | Billing surface | | ------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------ | | **Managed collector → CloudWatch OTLP** | CloudWatch-first; scrape-only today; label hygiene in place | Collector-hour + **$0.50/GB OTLP** (not per-series custom metrics) | | **Managed collector → AMP** | Grafana/AMP-native; recording rules matter | Collector-hour + AMP samples ingested | | **ADOT dual-export** | Same collector must export traces/logs + metrics with custom processors | Collector ops + AMP and/or X-Ray lines | | **Classic CloudWatch custom metrics / EMF** | Low series count; no `/metrics` exporters | $0.30/metric-month or EMF aggregation | Full destination matrix and cost CSV: [CloudWatch managed Prometheus collectors (July 2026)](https://www.joinhorizon.co/blog/cloudwatch-managed-prometheus-collectors-july-2026/). The managed scraper removes agent HA; it does **not** remove cardinality discipline. ## Tier 4: ADOT + AMP + Grafana, once the second backend is justified ![Architecture diagram of ADOT exporting traces to X-Ray and metrics to Amazon Managed Service for Prometheus](../../assets/images/blog/aws-observability-beyond-cloudwatch-otel-prometheus-grafana-2026-amp.webp) _Figure: the dual-export in the article. Traces go to X-Ray. Metrics go to Managed Prometheus for PromQL. Managed Grafana reads that store when you need one pane. The same scrape job does not also write a CloudWatch OTLP dataset. [Open the draw.io source](https://www.joinhorizon.co/examples/architecture-blog-2026/aws-observability-beyond-cloudwatch-otel-prometheus-grafana-2026/amp.drawio)._ Step up to **ADOT + Amazon Managed Service for Prometheus (AMP) + Amazon Managed Grafana (AMG)** when you can name the reason **and** managed collectors writing to CloudWatch OTLP will not close the gap: - You're **PromQL/Prometheus-native** with existing exporters and dashboards. - You have **high-cardinality** metrics where Prometheus's model beats CloudWatch custom metrics on both ergonomics and cost. - You want **OpenTelemetry-native, vendor-portable** instrumentation. - You need a **single Grafana pane** correlating AWS and third-party data. AMP is serverless and Prometheus-compatible (PromQL, Multi-AZ, EKS + self-managed K8s), with default 150-day retention configurable up to 3 years. AMG is fully managed Grafana over CloudWatch, X-Ray, Prometheus, and third-party sources. The pragmatic shape is **instrument once with OpenTelemetry and dual-export**: traces to X-Ray (feeding Application Signals' service map and SLOs) and metrics to AMP (for PromQL + high cardinality). That's exactly what the [ADOT collector config](https://www.joinhorizon.co/examples/architecture-blog-2026/observability-stack/adot-collector-config.yaml) does. The cost: one more component, the collector, to run and keep upgraded. > **What broke:** A team adopted AMP + AMG on day one for a new EKS platform "to do observability properly," scraping every exporter at 15s and granting all 12 engineers Grafana Editor. The first month's bill was dominated by AMP **ingestion** (the scrape-everything default) and inflated by paying $9/Editor for engineers who only ever viewed dashboards. Nothing was _wrong_; it just cost multiples of what it needed to. The fix was unglamorous: raise scrape intervals on non-alerting series, drop unused metric families at the source, and reassign 8 users to the $5 Viewer tier. The mistake was adopting the tools before measuring whether CloudWatch + Application Signals already answered the questions, then running them with no cost discipline. ## The cost lever that surprises people: ingestion, not retention AWS is explicit that **metric ingestion is the largest AMP cost driver**, and that **cutting retention rarely helps**. The [three levers](https://www.joinhorizon.co/examples/architecture-blog-2026/observability-stack/amp-cost-control.md), in order: 1. **Raise the scrape interval** on series that don't need 15s resolution (60s is ~4x fewer samples for those series). 2. **Filter metric families and high-cardinality labels at the source:** one runaway label (user ID, request ID, full URL) can multiply a series into millions. 3. **Pre-aggregate with recording rules:** compute the p99/error-rate once instead of scanning raw series on every dashboard load (also cuts query-sample cost). Leave retention alone unless compliance demands a change. For AMG, default users to **Viewer ($5)** and reserve **Editor ($9)** for dashboard authors. ## AWS services map | Need | Service | Skip when | | ----------------------------- | ---------------------------------------- | ----------------------------------------- | | Agentless Prometheus scrape | CloudWatch managed collectors (Jul 2026) | Need ADOT traces/logs on same collector | | OpenTelemetry pipeline | ADOT on EKS/ECS | Metrics-only scrape → managed collectors | | Prometheus-compatible storage | AMP or CloudWatch OTLP dataset | <5K series, CloudWatch metrics sufficient | | Dashboards + alerting | CloudWatch or Amazon Managed Grafana | One pane already answers SLO questions | ## What to do this week 1. Inventory which workloads have **Application Signals** on. Enable it (with **Transaction Search**) on your top revenue services before you consider any new backend. 2. For each AMP/AMG workload, ask: _what specific question does CloudWatch + Application Signals fail to answer?_ If you can't name it, you've found a candidate to retire. 3. Run the [tier decision matrix](https://www.joinhorizon.co/examples/architecture-blog-2026/observability-stack/observability-tier-decision-matrix.md) per workload. Don't apply one stack uniformly. 4. If you scrape Prometheus `/metrics` into CloudWatch today: read the [July 2026 managed collectors post](https://www.joinhorizon.co/blog/cloudwatch-managed-prometheus-collectors-july-2026/) and pick **one** destination per job before creating a scraper. 5. If you run AMP: apply the [cost-control levers](https://www.joinhorizon.co/examples/architecture-blog-2026/observability-stack/amp-cost-control.md): audit cardinality, raise scrape intervals, filter at source. 6. Audit Grafana licenses: reassign view-only engineers from Editor to Viewer. ## What this post doesn't cover - **CloudWatch alarm and Logs Insights fundamentals:** see [CloudWatch metrics, logs, and alarms best practices](https://www.joinhorizon.co/blog/aws-cloudwatch-observability-metrics-logs-alarms-best-practices/). - **Distributed-systems debugging workflow** (how to actually use traces in an incident): see [debugging production distributed AWS systems](https://www.joinhorizon.co/blog/debug-production-distributed-aws-systems/). - **A hands-on OpenTelemetry + chaos tutorial:** see [the OTel demo game post](https://www.joinhorizon.co/blog/otel-demo-game-aws-observability-chaos-engineering/). - **Amazon Managed Grafana workspace ops** (Grafana 12.4 upgrades, Viewer-default seats, NAC/VPC, CMK, service-account hygiene): see [Amazon Managed Grafana workspace best practices (2026)](https://www.joinhorizon.co/blog/amazon-managed-grafana-workspace-best-practices-2026/). - **Loki/log-analytics backends and Grafana OnCall:** out of scope here. - **Exact current pricing:** confirm AMP per-sample rates, managed collector-hour rates, CloudWatch OTLP $/GB, and AMG per-user rates on the respective AWS pricing pages; figures here are the mid-2026 model. - **Managed collector setup walkthroughs:** EKS IRSA and MSK/OpenSearch console flows live in AWS solution guides; see [CloudWatch managed Prometheus collectors (July 2026)](https://www.joinhorizon.co/blog/cloudwatch-managed-prometheus-collectors-july-2026/). --- **Related:** [CloudWatch managed Prometheus collectors (July 2026)](https://www.joinhorizon.co/blog/cloudwatch-managed-prometheus-collectors-july-2026/) · [CloudWatch observability best practices](https://www.joinhorizon.co/blog/aws-cloudwatch-observability-metrics-logs-alarms-best-practices/) · [Observability FinOps & cardinality cost control](https://www.joinhorizon.co/blog/aws-observability-finops-cardinality-cost-control/) · [CloudWatch logging costs](https://www.joinhorizon.co/blog/aws-cloudwatch-logging-costs-observability/) · [Debug production distributed systems](https://www.joinhorizon.co/blog/debug-production-distributed-aws-systems/) · [AWS managed services](https://www.joinhorizon.co/services/aws-managed-services/) **If you only do one thing:** Before standing up any new metrics backend, enable CloudWatch Application Signals with Transaction Search on your top services and ask what question it _fails_ to answer. If you can't name the gap, you don't need the second stack, and you've just avoided doubling the bill. ## More in This Track Part of the **Engineering Guides** library (June 2026). - Next: [Part 2](https://www.joinhorizon.co/blog/prometheus-cardinality-explosion-amp-cloudwatch-cost-control/) - Browse tracks: [Engineering Guides hub](https://www.joinhorizon.co/resources/engineering-guides/) ## Related reading - [The AWS CLI Bug That Broke /dev/null Across Your Entire System](https://www.joinhorizon.co/blog/aws-cli-chmod-dev-null-streaming-bug-2026/) - [AWS Environment Parity: Why Dev/Staging/Prod Drift Costs More Than It Saves](https://www.joinhorizon.co/blog/aws-environment-parity-dev-staging-production/) - [What DevOps Guides Don't Tell You About Production AWS](https://www.joinhorizon.co/blog/devops-exercises-aws-production-reality/) - [DevOps on AWS: CodePipeline vs GitHub Actions vs Jenkins](https://www.joinhorizon.co/blog/devops-on-aws-codepipeline-vs-github-actions-vs-jenkins/) - [Two Free LocalStack Alternatives in 2026: MiniStack vs floci](https://www.joinhorizon.co/blog/ministack-free-localstack-alternative-aws-emulator/) - [The Terraform Command Cheat Sheet for AWS Engineers (2026 Edition)](https://www.joinhorizon.co/blog/terraform-commands-cheat-sheet-aws-2026/) - [How to Build Ultra-Fast Asset Pipelines with Bun, Vite, and Rust-Based Tooling (2026)](https://www.joinhorizon.co/blog/ultra-fast-asset-pipelines-bun-vite-rust/) ## FAQ ### Do we need Amazon Managed Prometheus if we already use CloudWatch? Usually not, unless you are already Prometheus/PromQL-native or have high-cardinality metrics that get expensive as CloudWatch custom metrics. For most teams, CloudWatch core (metrics, logs, alarms, Logs Insights) plus CloudWatch Application Signals for APM covers application and AWS-service observability without a second backend to operate. Reach for Amazon Managed Service for Prometheus (AMP) when you have existing Prometheus exporters and Grafana dashboards, a heavy Kubernetes estate, or label-rich metrics where Prometheus cardinality handling and PromQL are better tools. Adopting AMP "to be complete" adds a second billing surface and a second query language for no measured benefit. Adopt it because you hit a specific gap. ### What is CloudWatch Application Signals and what setup does it need? Application Signals is the APM layer inside CloudWatch: it auto-discovers your services and dependencies, draws an application map, tracks Service Level Objectives (SLOs), and correlates traces, so you can go from a high-level fault/latency summary down to the offending span. It auto-instruments applications running on Amazon EKS, EC2, ECS, Kubernetes, Lambda, or on-premise, and it can ingest OpenTelemetry telemetry via AWS Distro for OpenTelemetry (ADOT) and the CloudWatch agent. One important setup note: you must enable Transaction Search to unlock the full APM feature set under the unified Application Signals pricing that includes X-Ray traces and transaction spans. For EKS specifically, recent versions of the CloudWatch Observability add-on can auto-monitor workloads with a single configuration flag. ### When should we NOT use Amazon Managed Grafana? Skip Amazon Managed Grafana (AMG) when CloudWatch dashboards already answer your questions and your data lives entirely in AWS. You do not need a separate visualization layer and its per-active-user billing. AMG earns its place when you need to correlate AWS telemetry with third-party sources in one pane, when your team already lives in Grafana, or when you want PromQL/Loki-style dashboards over AMP. Watch the pricing model: AMG bills per active user per workspace per month (Editor/Admin at $9, Viewer at $5 in the mid-2026 model), with optional Enterprise plugins adding $45 per active user. The common over-spend is making every engineer an Editor when most only view dashboards. Assign Viewer by default and Editor only to people who build dashboards. ### What actually drives Amazon Managed Service for Prometheus cost, and how do we cut it? AWS documents that metric ingestion is the largest cost driver for most AMP customers, not storage. The three levers that move the bill, in order: (1) increase the scrape interval for series that do not need second-level resolution (moving infra metrics from 15s to 60s is roughly 4x fewer samples for those series); (2) filter unused metric families and high-cardinality labels at the source before they are ingested; and (3) pre-aggregate repeated queries with recording rules, which also cuts query-sample cost. Reducing the retention period is explicitly called out by AWS as unlikely to help much, because storage is a minor component. Default retention is 150 days, configurable up to 3 years, so set it to your compliance need rather than treating it as a cost lever. ### Should we instrument with OpenTelemetry or use CloudWatch agent auto-instrumentation? Use OpenTelemetry (via ADOT) when you want vendor-portable instrumentation: the same SDK and collector can export to CloudWatch/X-Ray today and to a different backend later without re-instrumenting your code, which matters if you run hybrid or multi-cloud or want to avoid lock-in. Use CloudWatch agent auto-instrumentation / Application Signals auto-monitor when you want the fastest path to a service map and SLOs with the least configuration and you are committed to staying on CloudWatch. A common, pragmatic pattern is to instrument once with OpenTelemetry and dual-export: traces to X-Ray (feeding Application Signals) and metrics to Amazon Managed Service for Prometheus. See the ADOT collector config linked in this post. The trade-off is one more component (the collector) to run and keep upgraded. ### Is a three-tool stack (Application Signals + AMP + AMG) a reasonable target? It can be, but only when you have measured the need for each piece. The valid version is: Application Signals for application-level SLOs and traces, AMP for high-cardinality infrastructure metrics from a Prometheus-heavy Kubernetes estate, and AMG as the unified dashboard spanning AWS and third-party data. The cost of that completeness is two billing surfaces and two query languages (CloudWatch metrics + PromQL), plus the operational overhead of a collector fleet. The anti-pattern is adopting all three on day one because the reference architecture showed them. Start with CloudWatch + Application Signals, prove a specific gap (PromQL-native team, cardinality cost, third-party correlation), and add the next tier only to close that gap. ### When should we use CloudWatch managed Prometheus collectors instead of ADOT or AMP? On July 31, 2026 AWS shipped agentless CloudWatch managed Prometheus collectors that scrape /metrics from EKS, EC2, ECS (via Cloud Map), MSK, and OpenSearch and land metrics in a CloudWatch OpenTelemetry dataset at $0.50/GB ingested (15 months storage included), plus collector-hour fees on the AMP CreateScraper surface. Prefer managed collectors that write to CloudWatch when your primary pane is CloudWatch, you currently run a self-managed scrape fleet only for metrics, and you do not need ADOT processor flexibility for traces/logs on that path. Prefer AMP as the destination when you are Grafana/recording-rules native. Prefer classic CloudWatch custom metrics or EMF when series count is low and you are not Prometheus-exporter based. Do not dual-write the same scrape job to CloudWatch OTLP and AMP. See the managed collectors post for the destination matrix. --- *Source: https://www.joinhorizon.co/blog/aws-observability-beyond-cloudwatch-otel-prometheus-grafana-2026/*