---
title: Bloom Filters and HyperLogLog in Production on ElastiCache Redis
description: Bloom filters shave 90% of negative lookups; HyperLogLog estimates cardinality without storing every user ID. Redis modules on ElastiCache for abuse detection and feed deduplication.
url: https://www.joinhorizon.co/blog/bloom-filters-hyperloglog-production-elasticache/
datePublished: 2026-06-12T00:00:00.000Z
dateModified: 2026-10-28T00:00:00.000Z
author: palaniappan-p
category: Cloud Architecture
tags: engineering-guide, redis, elasticache, performance, aws
---

# Bloom Filters and HyperLogLog in Production on ElastiCache Redis

> Bloom filters shave 90% of negative lookups; HyperLogLog estimates cardinality without storing every user ID. Redis modules on ElastiCache for abuse detection and feed deduplication.

**ElastiCache Redis (June 2026)** supports **Bloom filters** (RedisBloom module where enabled) and **HyperLogLog** (`PFADD`/`PFCOUNT`) natively. HyperLogLog gives cheap cardinality for “how many unique IPs today?” without Prometheus label explosion.

## Symptom → mechanism → AWS control

| Production symptom                | Mechanism                  | AWS control                                           |
| --------------------------------- | -------------------------- | ----------------------------------------------------- |
| Origin hit on known-absent keys   | No negative caching        | Redis Bloom filter (RedisBloom module) on ElastiCache |
| Memory explosion on unique counts | Exact SET of user IDs      | HyperLogLog PFADD/PFCOUNT on ElastiCache              |
| False positive DB queries         | Bloom filter FPR not tuned | Size filter for 1% FPR, accept rare origin hits       |

**Opinionated take:** Use Bloom filters for 'probably not in DB' guards and HyperLogLog for analytics cardinality. Never use either when exact counts are a compliance requirement.

> **Benchmark pattern (hypothetical workload):** Bloom filter in ElastiCache Redis (1% FPR, 10M keys) uses 12MB vs 800MB for full set; HyperLogLog cardinality of daily unique visitors: 12KB memory, ±0.8% error vs exact SET at 240MB.

## Bloom filter

- **Use:** “Have we seen this dedup key?” with false positives OK
- **Avoid:** Need exact membership proof for billing

Configure expected items and false positive rate; size is fixed at creation.

## HyperLogLog

HyperLogLog uses ~12 KB per key for ~0.81% error on billions of uniques. That suits **dashboard UV** metrics, not invoicing.

## AWS patterns

| Use case        | Structure                                       |
| --------------- | ----------------------------------------------- |
| API abuse       | Bloom per IP window in Redis                    |
| Unique visitors | HLL per day, export to CloudWatch custom metric |
| Feed dedup      | Bloom + DynamoDB for positives                  |

## AWS services map

| Need                     | Service                           | Skip when                           |
| ------------------------ | --------------------------------- | ----------------------------------- |
| Probabilistic membership | ElastiCache Redis with RedisBloom | Exact membership required (billing) |
| Cardinality estimation   | ElastiCache HyperLogLog           | Need exact counts for invoicing     |
| CDN negative cache       | CloudFront custom error caching   | Dynamic per-user responses          |

## What to do this week

1. Replace `SET` of all session IDs with HLL for daily active count dashboard.
2. Add Bloom pre-check before expensive Aurora query.
3. Monitor Redis memory. Bloom is cheaper than full SET until false positive rate hurts UX.

For node-hour and Serverless ECPU rates, see the [ElastiCache pricing guide](https://www.joinhorizon.co/blog/amazon-elasticache-pricing-nodes-serverless-2026/).

## More in This Track

Part of the **Engineering Guides** library (June 2026).

- Previous: [Part 2](https://www.joinhorizon.co/blog/distributed-cache-invalidation-multi-level-caching-aws/)
- Browse tracks: [Engineering Guides hub](https://www.joinhorizon.co/resources/engineering-guides/)

## What this guide doesn't cover

General cache invalidation is covered in part 2 of the caching track.

## FAQ

### When should I not use a Bloom filter on ElastiCache?
Skip Bloom when membership must be exact: billing, entitlements, or any compliance count. False positives send rare origin hits, which is acceptable for abuse and feed dedup but not for invoices. Size the filter for about 1% FPR and accept those hits.

### When is HyperLogLog the wrong cardinality store?
Do not use PFADD/PFCOUNT when you must invoice unique IDs. HyperLogLog is for dashboard UV-style estimates (~12 KB, about 0.81% error). An exact SET of user IDs is the memory explosion this guide is trying to avoid.

### Does RedisBloom ship on every ElastiCache Redis cluster?
No. Bloom filters need the RedisBloom module where your engine and parameter group enable it. HyperLogLog (PFADD/PFCOUNT) is native Redis. Confirm module availability in the Region and engine version before you design around Bloom.

### What could go wrong if I treat CloudFront negative cache as a Bloom filter?
CloudFront custom error caching is for CDN miss patterns, not per-user membership. Dynamic per-user responses should not live in a shared negative cache. Use Redis Bloom for “probably not in DB” guards and DynamoDB for confirmed positives.

---

*Source: https://www.joinhorizon.co/blog/bloom-filters-hyperloglog-production-elasticache/*