---
title: How to Build a Serverless Data Pipeline with AWS Glue and Athena
description: Glue ETL + Athena SQL without servers, still the default lake path. July 2026: Glue 5.x jobs, Parquet partitions, ~$0.44/DPU-hour + $5/TB scan math, pipeline checklist.
url: https://www.joinhorizon.co/blog/how-to-build-serverless-data-pipeline-glue-athena/
datePublished: 2026-04-03T00:00:00.000Z
dateModified: 2026-10-28T00:00:00.000Z
author: palaniappan-p
category: Data & Analytics
tags: how-to-guide, glue, athena, etl, data-pipeline, aws
---

# How to Build a Serverless Data Pipeline with AWS Glue and Athena

> Glue ETL + Athena SQL without servers, still the default lake path. July 2026: Glue 5.x jobs, Parquet partitions, ~$0.44/DPU-hour + $5/TB scan math, pipeline checklist.

AWS Glue still owns the serverless ETL path: extract raw data (logs, CSVs, databases), transform (clean, enrich, aggregate), land curated files for analysis. Athena is the serverless SQL engine over S3 (+ Glue Data Catalog). As of **July 2026**, prefer a current **Glue 5.x** Spark job line where available, write **Parquet** with partitions, and treat Athena cost as **bytes scanned** ($5/TB list, verify region), not row count.

**First-party benchmark (illustrative, us-east-1 list rates):** Glue ~**$0.44/DPU-hour**; a 2-DPU × 30 min/day job ≈ **~$13/mo**; Athena at 50 × 100MB-effective scans (5 GB at $5/TB) ≈ $0.03. Athena only gets expensive when queries scan unpartitioned data, so partition and stop `SELECT *`.

**Reproduce this:** [Glue + Athena pipeline checklist](https://www.joinhorizon.co/examples/architecture-blog-2026/glue-athena-pipeline/glue-athena-pipeline-checklist.md)

> **Building Data Pipelines on AWS?** [AWS data services](https://www.joinhorizon.co/services/aws-data-analytics/) · [contact](https://www.joinhorizon.co/contact-us/).

## Step 1: Understand the Glue + Athena architecture

```
Raw Data (S3, RDS, APIs)
  ↓
AWS Glue Job (scheduled ETL)
  → Extract (read from source)
  → Transform (clean, dedupe, enrich)
  → Load (write to S3 in Parquet format)
  ↓
Athena (SQL queries on S3 data)
  → Run analytics
  → Generate reports
```

**Key components:**

- **Glue Crawler**: Auto-discovers schema from raw data
- **Glue Job**: Executes PySpark ETL code
- **Glue Data Catalog**: Metadata about your tables
- **Athena**: SQL engine for querying S3 data
- **Partitioning**: Organize S3 data by date/region for fast queries

## Step 2: Set up S3 and the Data Catalog

Create S3 buckets for raw data, transformed data, and Athena results:

```bash
# Raw data (source)
aws s3api create-bucket --bucket raw-data-bucket --region us-east-1

# Transformed data (processed)
aws s3api create-bucket --bucket transformed-data-bucket --region us-east-1

# Athena results (query output)
aws s3api create-bucket --bucket athena-results-bucket --region us-east-1

# Enable versioning (optional but recommended)
aws s3api put-bucket-versioning --bucket raw-data-bucket --versioning-configuration Status=Enabled
```

## Step 3: Crawl raw data with a Glue crawler

A Glue Crawler automatically discovers schema from your raw data:

1. Go to **AWS Glue** → **Crawlers**
2. Click **Create crawler**
3. **Crawler name**: `raw-data-crawler`
4. **Data sources**:
   - Type: S3
   - S3 path: `s3://raw-data-bucket/logs/`
5. **IAM role**: Create or select a role with S3 access
6. **Output configuration**:
   - Database: Create new database `raw_data`
   - Table prefix: `raw_` (tables will be named `raw_logs`, etc.)
7. **Scheduling**:
   - Frequency: Daily at 2 AM
   - (or manual for testing)
8. Click **Create crawler**

Run the crawler:

```bash
aws glue start-crawler --name raw-data-crawler
```

Check results in **Glue** → **Databases** → `raw_data` → **Tables**. The crawler will have created a table with schema inferred from your data.

## Step 4: Create a Glue ETL job

Create a Glue job to transform raw data:

1. Go to **AWS Glue** → **Jobs**
2. Click **Create job**
3. **Job name**: `transform-logs`
4. **Type**: Spark (for large datasets) or Python Shell (for small ones)
5. **IAM role**: Select the role with S3 access
6. **Script path**: `s3://glue-scripts/transform-logs.py`
7. Click **Create job**

Write the ETL script:

```python
# transform-logs.py
import sys
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.transforms import *
from pyspark.sql.functions import col, to_timestamp, year, month, day

args = sys.argv[1:]
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args[0], args)

# Read raw data from Glue Data Catalog
raw_logs = glueContext.create_dynamic_frame.from_table(
    database="raw_data",
    table_name="raw_logs"
)

# Convert to DataFrame for transformations
df = raw_logs.toDF()

# Transformations
df = df.filter(col("event_type").isNotNull())  # Remove nulls
df = df.withColumn("timestamp", to_timestamp(col("timestamp")))  # Parse date
df = df.dropDuplicates()  # Dedupe
df = df.withColumn("year", year(col("timestamp")))
df = df.withColumn("month", month(col("timestamp")))
df = df.withColumn("day", day(col("timestamp")))

# Write transformed data to S3 partitioned by date
df.repartition(col("year"), col("month"), col("day")).write \
    .mode("overwrite") \
    .parquet("s3://transformed-data-bucket/logs/")

job.commit()
```

Upload the script to S3:

```bash
aws s3 cp transform-logs.py s3://glue-scripts/
```

## Step 5: Run the Glue job on a schedule

1. In the Glue job, go to **Job details**
2. **Trigger**: Set to trigger daily at 3 AM (after crawler runs at 2 AM)
3. **Max retries**: 1
4. Click **Save**

Or trigger manually:

```bash
aws glue start-job-run --job-name transform-logs
```

Monitor the job:

```bash
aws glue get-job-run --job-name transform-logs --run-id xxxxx
```

## Step 6: Create an Athena table from transformed data

Once Glue writes transformed data to S3, create an Athena table to query it.

Option A: Glue Crawler (auto-detect schema)

Run a second crawler on the transformed data bucket:

```bash
aws glue start-crawler --name transformed-data-crawler
```

Option B: Manual Athena SQL

```sql
CREATE EXTERNAL TABLE IF NOT EXISTS transformed_logs (
    event_type STRING,
    user_id STRING,
    timestamp BIGINT,
    message STRING
)
PARTITIONED BY (
    year INT,
    month INT,
    day INT
)
STORED AS PARQUET
LOCATION 's3://transformed-data-bucket/logs/'
```

Add partitions:

```sql
ALTER TABLE transformed_logs ADD IF NOT EXISTS
    PARTITION (year=2026, month=4, day=3)
        LOCATION 's3://transformed-data-bucket/logs/year=2026/month=4/day=3/'
```

## Step 7: Query data with Athena

Go to **Amazon Athena** → **Query editor**:

```sql
-- Count events by type
SELECT event_type, COUNT(*) as count
FROM transformed_logs
WHERE year = 2026 AND month = 4
GROUP BY event_type
ORDER BY count DESC
```

```sql
-- Find events for a specific user
SELECT timestamp, event_type, message
FROM transformed_logs
WHERE user_id = 'user-12345' AND year = 2026
ORDER BY timestamp DESC
```

```sql
-- Daily aggregation
SELECT year, month, day, COUNT(*) as total_events
FROM transformed_logs
WHERE year = 2026
GROUP BY year, month, day
ORDER BY year, month, day
```

## Step 8: Optimize for cost and performance

### Partition strategy

Organize data in S3 with partitions so Athena only scans relevant data:

```
s3://transformed-data-bucket/logs/
├── year=2026/
│   ├── month=1/
│   │   ├── day=1/
│   │   ├── day=2/
│   │   └── ...
│   ├── month=2/
│   └── ...
└── year=2026/
```

Query with partition filters to reduce scans:

```sql
-- Scans ONLY April 2026 data
SELECT COUNT(*) FROM transformed_logs
WHERE year = 2026 AND month = 4

-- Scans ALL data (100x more expensive!)
SELECT COUNT(*) FROM transformed_logs
```

### Compress data

Use Parquet format with compression:

```python
# In Glue job
df.write.mode("overwrite").option("compression", "snappy").parquet("s3://bucket/data/")
```

Parquet + Snappy compression = 50-80% smaller than CSV/JSON.

### Use a columnar format

Query only required columns:

```sql
-- Efficient: scans only 3 columns
SELECT user_id, event_type, timestamp FROM logs

-- Inefficient: scans all columns
SELECT * FROM logs
```

## Step 9: Automate reporting

Create a scheduled Athena query that runs daily and saves results:

```python
# lambda_function.py
import boto3
from datetime import datetime

athena = boto3.client('athena')

def run_daily_report():
    query = """
    SELECT event_type, COUNT(*) as count
    FROM transformed_logs
    WHERE year = 2026 AND month = 4 AND day = 3
    GROUP BY event_type
    """

    response = athena.start_query_execution(
        QueryString=query,
        QueryExecutionContext={'Database': 'default'},
        ResultConfiguration={'OutputLocation': 's3://athena-results-bucket/'}
    )

    print(f"Query started: {response['QueryExecutionId']}")

def lambda_handler(event, context):
    run_daily_report()
    return {'statusCode': 200}
```

Schedule via CloudWatch Events:

```bash
aws events put-rule --name daily-athena-report --schedule-expression "cron(0 8 * * ? *)"
aws events put-targets --rule daily-athena-report --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123456789012:function:daily-report"
```

## Step 10: Production patterns

### Pattern 1: Multi-stage pipeline

```
Raw → Staging (light transformation)
    ↓
Staging → Curated (business logic)
    ↓
Curated → Analytics (final queries)
```

Each stage has its own Glue job and S3 location. This allows:

- Reusing staging data for multiple downstream jobs
- Easy debugging (inspect data at each stage)
- Auditing transformation steps

### Pattern 2: Quality checks

Add data quality validation in Glue jobs:

```python
# Count nulls
null_count = df.filter(col("user_id").isNull()).count()
if null_count > 0:
    print(f"WARNING: {null_count} rows with null user_id")
    # Send SNS alert

# Check for unexpected values
invalid_types = df.filter(~df.event_type.isin(['login', 'logout', 'click'])).count()
if invalid_types > 0:
    print(f"ERROR: {invalid_types} rows with invalid event_type")
    # Fail job
    sys.exit(1)

# If we reach here, data is valid
df.write.mode("overwrite").parquet("s3://bucket/data/")
```

### Pattern 3: Incremental processing

Avoid reprocessing the same data:

```python
# Only process data from last 24 hours
from datetime import datetime, timedelta

cutoff_time = (datetime.now() - timedelta(days=1)).timestamp()
df = df.filter(col("timestamp") > cutoff_time)
```

This reduces processing time and cost on large datasets.

## Common mistakes to avoid

1. **Not partitioning data**
   - Unpartitioned: Every query scans all data (expensive)
   - Partitioned: Queries scan only relevant data (cheap)
   - Always partition by date, region, or other common filters

2. **Using CSV instead of Parquet**
   - CSV: Slow, uncompressed, row-oriented
   - Parquet: Fast, compressed, columnar
   - Convert to Parquet in Glue jobs

3. **Not adding partition metadata**
   - Athena needs partition info to optimize queries
   - Use `MSCK REPAIR TABLE` to add partitions:
     ```sql
     MSCK REPAIR TABLE my_table
     ```

4. **Querying without WHERE clauses**
   - `SELECT * FROM logs` scans all data
   - Always add partition filters: `WHERE year=2026 AND month=4`

5. **Ignoring job failures**
   - Set up SNS alerts for failed Glue jobs
   - Monitor job logs in CloudWatch

## Cost estimation

For 1GB of raw data ingested daily:

| Component                                     | Cost         |
| --------------------------------------------- | ------------ |
| Glue (2 DPUs, 30 mins/day)                    | ~$13/month   |
| S3 storage (30GB/month retention)             | ~$0.69/month |
| Athena (50 queries/month, 100MB scanned each) | ~$0.03/month |
| **Total**                                     | ~$14/month   |

## What this post doesn't cover

Iceberg/S3 Tables lakehouse evolution and dbt-vs-Glue ownership are covered in the lakehouse and Glue-vs-dbt companions. Athena Capacity Reservations for steady BI are optional cost controls beyond this how-to.

## What to do Monday morning

1. Stand up raw / curated / athena-results buckets + one Glue Catalog database.
2. Convert a sample day to Parquet with partition keys matching your filters.
3. Run an Athena query **with** partition predicates; note data scanned in query stats.
4. Walk the [pipeline checklist](https://www.joinhorizon.co/examples/architecture-blog-2026/glue-athena-pipeline/glue-athena-pipeline-checklist.md).
5. Next: [building a data lake](https://www.joinhorizon.co/blog/building-a-data-lake-on-aws-s3-glue-athena-architecture/) · [Glue vs dbt](https://www.joinhorizon.co/blog/aws-glue-vs-dbt-on-aws-data-transformation-guide/).
6. [Talk to FactualMinds](https://www.joinhorizon.co/contact-us/) · [data analytics services](https://www.joinhorizon.co/services/aws-data-analytics/).

## FAQ

### What is the difference between AWS Glue and Lambda for ETL?
Lambda is compute: you write code and it runs. Glue is a fully managed ETL service with built-in job scheduling, error handling, and data catalog features. For simple transformations (1-2 scripts), Lambda is faster to implement. For complex ETL pipelines (10+ transformations, scheduling, data lineage), Glue is easier: you define jobs in the console, Glue handles retries and logging. Glue also includes the Data Catalog (metadata about your data), which Lambda doesn't provide.

### How much does Glue cost?
Glue charges about $0.44 per DPU-hour in us-east-1 (verify region). A 10-minute job using 2 DPUs ≈ 0.33 hours × 2 × $0.44 ≈ $0.29. Athena list pricing is about $5 per TB scanned (verify region). A 100GB effective scan ≈ $0.50. Partitioning + Parquet usually dominate the bill more than the list rate.

### How do I partition data for better Athena performance?
Partition Athena tables by date, region, or other common filters. Instead of one table with 1M rows, partition by year/month so Athena only scans relevant partitions. Example: logs/year=2026/month=04/day=03/data.parquet. Athena scans only data matching your query (e.g., "WHERE year=2026 AND month=04"). Without partitioning, Athena must scan all 1M rows. With partitioning, it scans 10K rows. Cost savings: 100x.

### Can I use Glue with data in real-time?
Classic Glue Spark jobs are batch (hourly/daily schedules). For sub-minute paths use Kinesis/Firehose/Flink/Lambda, or evaluate Glue streaming ETL where it fits. Do not force a nightly batch job into a real-time SLA. DMS remains the usual CDC path into the lake.

### What format should I use for Athena queries — Parquet, CSV, or JSON?
Parquet is best for analytics: (1) Columnar format (Athena only reads relevant columns), (2) Compressed (50-80% smaller than CSV), (3) Schema-aware (data types preserved). CSV/JSON are row-based and less efficient. For 1GB of data: CSV = 1GB scanned, Parquet = 200-400MB scanned. Glue can convert CSV to Parquet as a transformation step, so raw data can be CSV and Glue converts to Parquet for analytics.

---

*Source: https://www.joinhorizon.co/blog/how-to-build-serverless-data-pipeline-glue-athena/*