Streaming Interview Guide

The #1 most-asked messaging question in architect interviews — knowing when to use which service:

Decision Framework

• "I need to decouple two services" → SQS
• "One event, multiple consumers" → SNS (simple) or EventBridge (with content filtering)
• "I need ordering + replay + high throughput" → Kinesis Data Streams
• "I need smart event routing with filtering" → EventBridge
• "I need the Kafka ecosystem" → Amazon MSK

Kinesis is the core data streaming service on AWS — understand the difference between Streams and Firehose:

Kinesis Data Streams vs Firehose

Shard Architecture

• Each shard: 1 MB/s write, 2 MB/s read (or 5 reads/sec)
• Enhanced Fan-Out: 2 MB/s per consumer per shard (dedicated throughput). Use when you have 3+ consumers.
• Partition Key: Determines which shard gets the record. Use high-cardinality keys (user_id) to avoid hot shards.
• On-Demand Mode (2021): Auto-scales shards, no capacity planning. Pay per GB. Best for unpredictable traffic.

Amazon MSK is managed Apache Kafka on AWS — choose it when you need the Kafka ecosystem:

MSK vs Kinesis — Decision Framework

MSK Serverless vs Provisioned

• MSK Serverless: Auto-scales, no broker management, pay per data. Best for variable workloads and teams that don't want to manage Kafka infrastructure.
• MSK Provisioned: Choose broker instances and config. Better price at sustained high throughput. Needed for advanced Kafka features.

Architecture patterns for processing streaming data at scale:

Lambda Architecture vs Kappa Architecture

Processing Options on AWS

Windowing Strategies (For Flink)

• Tumbling Window: Fixed size, non-overlapping (e.g., "count per 5-minute block"). Simplest.
• Sliding Window: Overlapping (e.g., "trending in last 15 min, updated every 1 min"). Most common for real-time dashboards.
• Session Window: Activity-based (e.g., "user session ends after 30 min of inactivity"). Used for user behavior analytics.

AWS architecture for live video streaming at scale — understanding this shows you can design media-heavy systems:

Architecture Flow (5 Steps)

1. Ingest — Live video source (camera, encoder) sends an RTMP/SRT stream to AWS Elemental MediaLive.
2. Transcode — MediaLive transcodes the stream into multiple bitrates and resolutions (ABR — Adaptive Bitrate): e.g., 1080p, 720p, 480p, 240p for different devices and network conditions.
3. Package — AWS Elemental MediaPackage packages the transcoded stream into HLS/DASH formats and provides origin endpoints with DVR/time-shift/catch-up TV capabilities.
4. Deliver — Amazon CloudFront (CDN) distributes the live stream globally to millions of viewers with low latency via 450+ edge locations.
5. Play — Viewers watch on web, mobile, or smart TV using a video player that dynamically switches quality based on bandwidth (ABR).

MediaLive vs IVS — When to Use Which

Streaming Protocols (Quick Reference)

• HLS (HTTP Live Streaming) — Apple standard. Works everywhere. 10-30s latency. Use for broad compatibility.
• DASH (Dynamic Adaptive Streaming over HTTP) — Open standard. Similar to HLS. Used by YouTube, Netflix.
• LL-HLS (Low Latency HLS) — Apple's low-latency extension. 2-4 seconds. Use for near-real-time.
• WebRTC — Sub-second latency. Use for video conferencing, not broadcast.

Server-side ad insertion (SSAI) is the enterprise approach to monetizing live streams while defeating ad blockers:

SSAI Architecture Flow

1. Live Stream + Ad Markers — MediaLive inserts SCTE-35 markers (cue points) into the video stream at designated ad break positions.
2. Ad Decision Server (ADS) — When a viewer requests the stream, AWS Elemental MediaTailor detects SCTE-35 markers and calls the Ad Decision Server to fetch personalized ads for that specific viewer.
3. Server-Side Stitching — MediaTailor transcodes the ad to match the stream's bitrate/format and stitches it directly into the video stream server-side. The viewer receives a single, seamless stream.
4. Content Delivery — CloudFront delivers the personalized stream (with stitched ads) to each viewer. Each viewer may see different ads.

SSAI vs Client-Side Ads (CSAI)

SCTE-35 Markers — What They Are

SCTE-35 is the industry standard for signaling ad breaks in video streams. MediaLive can insert them automatically on a timer, or your automation system can trigger them via the MediaLive API at specific moments (e.g., between game quarters).

Streaming questions cover both data streaming (Kinesis, MSK) and video streaming architecture. These test real-time data pipeline design.

1. Compare SQS, SNS, Kinesis Data Streams, and EventBridge. You need to process clickstream data from a website at 50,000 events/second with ordering guarantees. Which do you choose and why?
   Answer Guide

   Kinesis Data Streams — ordered within shard (by partition key), handles massive throughput, 7-day retention. SQS has no ordering (unless FIFO, which caps at 3,000 msg/s). SNS is pub/sub (no retention). EventBridge is for event routing, not high-throughput data streaming.
2. When would you choose Amazon MSK (Managed Kafka) over Kinesis Data Streams? A team already using open-source Kafka on-prem wants to migrate to AWS.
   Answer Guide

   MSK if: existing Kafka expertise, need Kafka ecosystem (Kafka Connect, Kafka Streams, ksqlDB), complex consumer group patterns, longer retention (unlimited). Kinesis if: fully serverless, tight AWS integration (Lambda, Firehose), simpler operations. Migration path: MSK preserves Kafka APIs, no code changes.
3. Design a real-time analytics pipeline for an e-commerce platform. You need to: track user behavior, calculate trending products (last 15 minutes), and populate a real-time dashboard.
   Answer Guide

   Clickstream → Kinesis Data Streams (partitioned by user_id) → Managed Apache Flink (sliding window aggregation over 15 min) → DynamoDB/ElastiCache for trending products → API Gateway + Lambda for dashboard API. Alternative: Kinesis → Firehose → S3 for batch analytics with Athena.
4. Your Kinesis stream has 10 shards. One shard is receiving 80% of all traffic because events are partitioned by a single popular product_id. How do you fix this hot shard problem?
   Answer Guide

   Use a composite partition key (product_id + random_suffix) to distribute across shards. Or use a hash-based partition key. Trade-off: you lose strict ordering for that product_id. Alternative: re-shard (split the hot shard), but this is temporary if the key distribution doesn't change.
5. A financial services company needs exactly-once processing semantics for their transaction stream. Can Kinesis provide this? How would you guarantee no duplicates and no data loss?
   Answer Guide

   Kinesis provides at-least-once delivery. For exactly-once: use KCL (Kinesis Client Library) with checkpointing + DynamoDB for deduplication (idempotency key per record). Use enhanced fan-out for dedicated throughput per consumer. Discuss: Kafka with transactions can provide exactly-once semantics natively — trade-off consideration.
6. You need to deliver live video to 100,000 concurrent viewers with sub-5-second latency. Would you use MediaLive + MediaPackage + CloudFront, or Amazon IVS? Justify your choice.
   Answer Guide

   IVS for simplicity (managed, sub-3s latency out-of-the-box, WebRTC-based). MediaLive stack for control (custom encoding profiles, DVR, DRM, SSAI ads, multi-CDN). At 100K viewers, IVS is simpler but less customizable. If you need ad insertion or premium features, go MediaLive stack.
7. Design a log aggregation pipeline for an application generating 10GB of logs per hour across 200 EC2 instances. Logs need to be searchable within 30 seconds of generation.
   Answer Guide

   CloudWatch Agent on instances → CloudWatch Logs → Subscription Filter → Kinesis Data Firehose → OpenSearch (near-real-time indexing, ~30s). Alternative: Fluent Bit → Kinesis Data Streams → Lambda/Flink → OpenSearch. Discuss: Firehose buffers (60-900 seconds), so tune buffer interval to 60s for near-real-time.
8. Explain Kinesis Data Firehose vs Kinesis Data Streams. A team asks: "Why would I use Firehose when Streams does the same thing?"
   Answer Guide

   They solve different problems. Streams: real-time processing, custom consumers, replay, ordering. Firehose: fully managed delivery to S3/Redshift/OpenSearch with built-in transformation (Lambda), batching, compression, and format conversion (Parquet). Use Streams for real-time processing, Firehose for data delivery/ETL.
9. Your streaming pipeline experiences a 2-hour outage. When it recovers, how do you reprocess the missed data without duplicating records that were already processed?
   Answer Guide

   Kinesis retains data for 24-365 hours. Use TRIM_HORIZON or AT_TIMESTAMP to replay from before the outage. Idempotent consumers (dedup using DynamoDB conditional writes) ensure no duplicates. For Firehose: check S3 for already-delivered files. For Kafka/MSK: reset consumer group offset to the outage timestamp.
10. A media company wants to monetize their live stream with personalized ads. Each viewer should see different ads based on their profile. How does server-side ad insertion (SSAI) work and why is it preferred over client-side?
    Answer Guide

    SSAI (MediaTailor) stitches ads into the video stream server-side — ad blockers can't detect them, seamless quality matching, frame-accurate insertion. CSAI relies on client player to fetch ads — blocked by ad blockers, buffering during transitions. SSAI uses SCTE-35 markers in the manifest to identify ad break positions.