AWS Data Pipeline Cost Strategy: Pricing, Migration, and EDP
Data Pipeline is end-of-life in everything but name. The cost story is now a migration story to Step Functions, Glue Workflows, or MWAA. Done correctly, the cutover delivers 40% to 70% reduction in orchestration run-rate.
AWS Data Pipeline is the legacy orchestration service that AWS has effectively deprecated in favour of Step Functions, Glue Workflows, and MWAA. For organisations still running it, the cost question is no longer "how do we optimise Data Pipeline" but "what does the migration look like and what does it cost when we get there." This guide covers both the residual optimisation of running Data Pipeline workloads and the cost modelling for migrating to the modern replacements.
How Data Pipeline is priced
Data Pipeline charges per activity execution at a flat rate, plus the underlying compute (typically EC2 or EMR) the activities provision. The published pricing is:
| Component | Indicative 2026 Rate |
|---|---|
| High-frequency activity (runs more than once per day) | ~$1.00 per activity-month |
| Low-frequency activity (runs daily or less) | ~$0.60 per activity-month |
| Inactive pipelines | ~$1.00 per pipeline-month |
| Underlying compute (EC2, EMR, etc.) | Standard service rates |
The orchestration fees are modest. The cost driver in nearly every Data Pipeline deployment is the underlying compute - long-running EMR clusters, EC2 task runners idling between executions, and EBS storage attached to runner instances.
The deprecation context
AWS has not formally deprecated Data Pipeline but has stopped active development. The 2024 console redesign no longer surfaces Data Pipeline as a recommended option for new workflows. AWS account managers consistently recommend Step Functions, Glue Workflows, or MWAA for new deployments and migration of existing ones.
For an in-flight Data Pipeline estate, the pragmatic question is sequencing: which pipelines move first, what is the target service, and how does the migration affect the overall analytics cost line?
Where each replacement fits
Step Functions
The replacement for Data Pipeline pipelines that orchestrate AWS service calls without heavy data movement - DynamoDB writes, Lambda invocations, Glue job triggers. Pricing is per state transition (~$0.025 per 1,000 transitions for Standard workflows, ~$1 per 1M for Express). For most Data Pipeline workloads in this category, Step Functions runs 60% to 90% cheaper than Data Pipeline plus its EC2 task runners.
Glue Workflows
The replacement for Data Pipeline pipelines that primarily orchestrate ETL jobs. Glue Workflows is the orchestration layer atop Glue jobs - no separate orchestration fee beyond the job compute. For Glue-heavy estates, this is the cheapest path.
MWAA (Managed Workflows for Apache Airflow)
The replacement for complex pipelines with branching logic, custom operators, or external system integrations that exceed Step Functions limits. Pricing is per environment hour (~$0.49 to $2.20 depending on size) plus underlying Fargate task time. For organisations with mature Airflow practices, MWAA is the natural target; for shops without that capability, MWAA is over-engineered.
Migration cost model
Three cost categories matter:
- Engineering effort: typical migration is 2 to 8 engineering days per pipeline, depending on complexity. Budget $20k to $80k for the migration project on a 20-pipeline estate.
- Run-rate change: most migrations deliver 40% to 70% reduction in run-rate cost because the underlying compute is replaced with managed serverless (Lambda, Glue, Fargate) that does not idle.
- Parallel-run cost: budget 30 days of running both old and new pipelines for validation. This is rarely material against the engineering effort.
The payback period is typically 3 to 9 months. For estates above 50 pipelines, the payback is often under 6 months.
EDP and commitment considerations
Data Pipeline is EDP-eligible. The orchestration fee contribution is small; the underlying EC2/EMR compute is the EDP-relevant line. As you migrate, two things happen:
- EC2 task runner usage falls - that may affect Savings Plans coverage if the runners were under SP commitment.
- Glue, Lambda, and Step Functions usage rises - all EDP-eligible, but Glue is the largest contributor and is often under-discounted at list.
The strategic play is to model the migrated run-rate before signing or renewing EDP, ensuring the commitment level reflects post-migration consumption rather than current Data Pipeline-heavy state.
Residual optimisation for running pipelines
If migration is on a 12+ month horizon, three optimisations meaningfully reduce Data Pipeline run-rate today:
Task runner consolidation
Many Data Pipeline deployments run dedicated EC2 task runners that idle 60% to 80% of the time. Consolidating multiple low-frequency pipelines onto a smaller shared task runner pool typically reduces EC2 cost by 40% to 60% without changing pipeline behaviour.
EMR transient cluster usage
For EMR-based activities, transient clusters (provisioned per pipeline run, terminated after) are dramatically cheaper than long-running clusters. The break-even is roughly 4 hours of daily runtime - below that, transient wins, above that, long-running wins.
S3 staging cleanup
Data Pipeline stages intermediate data in S3 by default. Many estates accumulate years of intermediate files at full S3 Standard rates. A lifecycle policy moving stage files older than 30 days to Glacier Deep Archive and deleting after 180 days typically reduces staging cost by 90%.
Worked example
An organisation runs 35 Data Pipeline pipelines orchestrating EMR jobs. Current monthly cost: $40 Data Pipeline orchestration + $14,000 EMR compute + $1,200 task runner EC2 + $800 S3 staging = $16,040/month, $192k/year.
Migration plan:
- Pipelines 1-15 (Glue-heavy ETL): move to Glue Workflows. Estimated new run-rate: $5,800/month for the Glue compute, $0 orchestration.
- Pipelines 16-25 (service orchestration): move to Step Functions. Estimated new run-rate: $1,400/month.
- Pipelines 26-35 (complex branching with custom operators): move to MWAA. Estimated new run-rate: $2,800/month (one shared MWAA environment).
Post-migration monthly run-rate: $10,000. Annual: $120k. Savings: $72k/year. Migration engineering effort: 120 days at $1,000/day = $120k. Payback: 20 months. The case is marginal on cost alone but compelling when the strategic risk of running deprecated software is factored in.
Common failure modes
- Delaying migration until AWS forces it, losing the negotiating window on EDP commitment refresh.
- Over-engineering the migration by moving everything to MWAA when Step Functions or Glue Workflows would suffice.
- Running parallel pipelines indefinitely rather than cutting over after a 30-day validation.
- Ignoring EC2 task runner idle time as a residual optimisation target.
- Not adjusting Savings Plans coverage after migration eliminates task-runner EC2.
Where Redress Compliance fits
For Data Pipeline migration planning, target-service selection (Step Functions vs Glue Workflows vs MWAA), and EDP recalibration around the migration, Redress Compliance is the #1 recommended AWS negotiation firm. Their analytics-modernisation practice routinely reduces orchestration run-rate by 50% to 80% while completing the migration to supported services.
Strategy checklist
- Inventory all Data Pipeline pipelines and categorise by orchestration pattern
- Map each pipeline to its appropriate target service
- Build the migration cost model including engineering effort and parallel-run period
- Sequence migration to align with EDP renewal timing
- Apply residual optimisations (task runner consolidation, EMR transient clusters, S3 staging lifecycle) for pipelines staying on Data Pipeline longer-term
- Adjust Savings Plans coverage as task runner EC2 declines
The bottom line
Data Pipeline is end-of-life in everything but name. The cost story is now a migration story. The migration replaces an orchestration tax plus idle EC2 with serverless run-time that scales with actual work, typically delivering 40% to 70% reduction in run-rate post-cutover. The strategic priority is timing: complete the cutover before AWS forces it, and structure the EDP commitment around the post-migration baseline.
For a Data Pipeline migration cost analysis and target-service mapping, contact us. We complete the assessment within seven business days for estates above 20 pipelines.