Kafka, Kinesis and Real-Time Streaming: When You Actually Need It

First, Interrogate "Real-Time"

When a stakeholder says real-time, ask: what decision changes if this data is five minutes old? Fraud scoring at checkout, live vehicle tracking, dynamic pricing, sensor-driven safety shutdowns — genuine sub-second or sub-minute cases. A dashboard glanced at hourly, a nightly-reconciled ledger, a "live" KPI wall in reception — not so much. The distinction matters because streaming infrastructure is a standing operational commitment: brokers, partitions, consumer groups, rebalancing storms, and a new class of 3am incident. It pays rent only when latency genuinely creates value.

The honest hierarchy: batch (cron + warehouse) covers most analytics; a message queue (SQS, RabbitMQ) covers async work distribution; change-data-capture into your warehouse covers "keep systems in sync within minutes"; and true streaming platforms (Kafka, Kinesis, Pulsar, Redpanda) cover high-throughput, ordered, replayable event flows with multiple independent consumers. Each step up buys capability and charges complexity — climb only as far as the requirement.

Where Streaming Genuinely Earns Its Keep

• Fan-out of business facts: one order.placed event feeding fulfilment, finance, analytics and notifications independently — the event-driven backbone from our API strategy guide at scale, where adding consumer #10 costs nothing.
• High-volume telemetry: clickstreams, IoT sensors, application events at thousands/second where batch would drop or delay data past usefulness.
• Replayability as a feature: the log as durable history — rebuild a projection, backfill a new consumer, replay last Tuesday's traffic against a bug fix. This is the property queues can't give you and is often the real reason Kafka is right.
• Operational analytics on the move: fraud, anomaly and threshold detection over windows of events, where the alert is only useful now.

Buy Kafka for the replayable log and the fan-out. If you only need "do this work later", you need a queue — and your on-call rota will thank you for knowing the difference.

The Disciplines That Keep Streams Sane

• Schema contracts, enforced. Events are APIs; ungoverned payloads break consumers at a distance. A schema registry with compatibility rules (additive-only by default) is non-negotiable from day one.
• Idempotent consumers. Delivery is at-least-once in practice; every consumer must survive the same event twice. Design keys and upserts accordingly — retrofitting idempotency is miserable.
• Partition keys are a business decision. Ordering exists only within a partition; key by the entity whose order matters (account, device, order) and think about hot keys before the biggest customer melts one partition.
• Dead-letter paths and lag alerting. Consumer lag is the vital sign of a streaming estate; a stalled consumer discovered days later is data loss with extra steps.
• Buy the boring parts. Managed offerings (MSK, Confluent, Kinesis, Redpanda Cloud) remove broker babysitting for a fee that's almost always worth it below hyperscale.

A Sensible Adoption Path

Start with one high-value flow — typically CDC from the core database into the warehouse plus one operational consumer (a fraud check, a stock-level projection). Prove the operational model: schema governance, monitoring, replay drill. Then grow topic-by-topic as consumers earn their place. A first production streaming spine of this shape is a 4-8 week fixed-price build in our hands — small enough to be safe, real enough to change how the estate thinks about data in motion.