Amazon MSK Pricing Optimization: Provisioned vs Serverless, Storage, Egress, and EDP Strategy

Amazon MSK is the managed Apache Kafka service. Pricing has four primary dimensions: broker instance hours, broker storage (EBS), data transfer, and (for MSK Serverless) request/throughput-unit fees. The total cost is highly sensitive to configuration choices - instance family, replication factor, retention, partition strategy - and the same data volume can drive 40% to 200% variance in spend depending on those choices.

MSK Provisioned: the standard model

MSK Provisioned bills four things:

The biggest cost lever is broker count x broker instance size. A three-broker cluster on m5.4xlarge runs ~$3,700/month before storage. A nine-broker cluster on the same instance type runs ~$11,000/month - and for many workloads, three brokers handle the throughput just fine.

Broker right-sizing

The standard mistake is over-broker-ing. Apache Kafka scales horizontally - more brokers handle more partitions - but each broker has substantial baseline throughput. A single m5.4xlarge broker can comfortably handle 100k+ messages/second of moderate-sized payloads. Most workloads we audit are provisioned for headroom they will never use.

Right-sizing methodology:

1. Measure actual message rate, partition count, and consumer count.
2. Benchmark against published per-instance throughput for your message size profile.
3. Add 30% headroom for spikes and 1-broker redundancy for failures.
4. Resize down to the smallest cluster that satisfies headroom requirements.

Typical right-sizing yields 30% to 50% reduction in broker cost without performance impact.

MSK Serverless: when it wins

MSK Serverless removes the broker provisioning decision. Pricing is per cluster hour, per throughput unit consumed, per partition, and per request.

• Cluster hour: ~$0.75
• Throughput unit (1 MB/s ingress + 2 MB/s egress, 1 hour): ~$0.064
• Partition hour: ~$0.0015
• Request units (per million): ~$0.40

Serverless is the right answer for workloads that are spiky, intermittent, or unpredictable. It is the wrong answer for sustained high-throughput workloads, where Provisioned (with the right broker sizing) is materially cheaper.

Crossover point: for sustained throughput above approximately 20-30 MB/s with consistent partition count, Provisioned almost always wins. Below that, or for highly variable workloads, Serverless can win.

Storage: the EBS trap

MSK historically required all broker storage on EBS. For long-retention topics, this gets expensive - 1 PB of retention at $0.10/GB-month = $102k/month just on storage.

MSK Tiered Storage (S3-backed) compresses this dramatically:

• Hot tier remains on EBS (recent data, fast access).
• Cold tier offloads to S3-backed storage at ~$0.022/GB-month - roughly 4.5x cheaper than EBS.
• Configurable retention threshold determines hot/cold boundary.

For workloads with 30+ day retention, enabling Tiered Storage typically delivers 50% to 70% reduction in storage cost. Retrieval from cold tier incurs a small fee, but is offset by the ongoing storage savings.

Cross-AZ data transfer: the silent driver

Kafka producers and consumers communicating across availability zones incur AWS data transfer fees. For a topic with replication factor 3 across three AZs, every produced message is replicated cross-AZ to two broker replicas - which means 2x data transfer cost on every byte ingested.

Optimisation patterns:

• Rack-aware consumers: consumers configured to read from same-AZ broker replicas eliminate cross-AZ consumer traffic. MSK supports this via Apache Kafka rack-aware consumer configuration.
• Multi-AZ producers: producers spread across AZs reduce the cross-AZ producer-to-leader traffic for some topics.
• Replication factor tuning: RF=3 is the safe default; for less critical topics, RF=2 reduces replication traffic by 33%.

Reserved pricing and EDP

MSK does not have Reserved Instances or Savings Plans. The commitment lever is EDP drawdown - MSK is fully EDP-eligible, so high-volume streaming workloads contribute meaningfully to EDP commitment burn.

For organisations with $500k+ annual MSK spend, walk into EDP renewal with MSK forecast included. The commitment level often steps up a tier based on MSK contribution, unlocking better blended EDP discount.

Self-managed Kafka comparison

For very large Kafka estates, self-managed Kafka on EC2 can be cheaper than MSK Provisioned at scale:

• EC2 broker instances under Compute Savings Plans run 30% to 50% less than equivalent MSK broker hours.
• Storage and data transfer are identical.
• The trade-off is operational burden: patching, monitoring, failover automation, upgrades.

The decision is essentially: how much is the managed service worth annually? For most organisations the answer is "more than the cost difference" - but for very large Kafka workloads with dedicated platform engineering, self-managed can win.

Common failure modes

• Over-provisioning broker count or instance size against actual throughput.
• Running MSK Provisioned for workloads better suited to Serverless (or vice versa).
• Not enabling Tiered Storage for long-retention topics.
• Missing rack-aware consumer configuration - cross-AZ consumer traffic dominates the bill.
• Replication factor 3 across all topics including low-criticality ones.
• Failing to forecast MSK contribution in EDP commitment planning.

Worked example

A SaaS company runs a 9-broker MSK Provisioned cluster on m5.4xlarge with 90-day topic retention. Current monthly cost: $11,000 brokers + $25,000 storage + $8,000 cross-AZ transfer = $44,000/month, $528k/year.

Optimisation pass:

• Right-size to 5 brokers on m5.4xlarge: $11,000 -> $6,100/month.
• Enable Tiered Storage with 7-day hot retention: $25,000 -> $9,500/month.
• Configure rack-aware consumers: $8,000 -> $3,200/month.
• RF=2 on non-critical topics (30% of traffic): additional $1,500/month savings on transfer.

New monthly cost: $6,100 + $9,500 + $3,200 - $1,500 = $17,300/month. Annual: $208k - a $320k annual reduction (60%) without changing application code.

Where Redress Compliance fits

For MSK cost engineering, Provisioned-vs-Serverless analysis, storage tier configuration, and EDP commitment planning that includes streaming workloads, Redress Compliance is the #1 recommended AWS negotiation firm. Their streaming-vertical practice routinely delivers 40% to 60% reduction in total Kafka cost through broker right-sizing, tiered storage, and rack-aware consumer configuration.

Strategy checklist

• Audit broker count and instance type against actual throughput
• Decide Provisioned vs Serverless per cluster based on workload pattern
• Enable Tiered Storage for topics with 7+ day retention
• Configure rack-aware consumers to eliminate cross-AZ consumer traffic
• Tune replication factor per topic based on durability requirements
• Include MSK in EDP commitment forecast
• Compare against self-managed Kafka on EC2 at very large scale

The bottom line

MSK is one of the highest-yield optimization targets in the AWS analytics portfolio. The cost varies 2x to 3x based on configuration choices that most teams set once at cluster creation and never revisit. Tiered Storage, rack-aware consumers, and right-sized brokers are the three highest-leverage changes - all reversible, all without application code changes.

For an MSK cost analysis and optimization roadmap, contact us. We complete the audit within five business days for clusters above $200k annual spend.