Real-Time Streaming Data Platform

A personal reference implementation of a streaming data platform: Kafka for ingestion, Apache Flink for stateful stream processing, and a Hudi-managed data lake for curated, queryable output.

Batch-only pipelines introduce latency that isn't acceptable for use cases like operational dashboards or near-real-time analytics. At the same time, naive streaming implementations often skip the details that make streaming trustworthy: exactly-once semantics, schema enforcement, and recovery from failure. The goal was to build a streaming pipeline that treats correctness and recoverability as first-class requirements, not afterthoughts.

• Must recover cleanly from task-manager restarts without duplicating or losing events.
• Must validate event schemas at ingestion time rather than failing silently downstream.
• Infrastructure must be fully reproducible from Terraform, with no manually configured resources.
• Local development must be possible without a live cloud environment, using Docker Compose.

• Designed the end-to-end data flow from Kafka topic to curated lake table.
• Implemented Flink jobs, including keyed state and checkpointing configuration.
• Defined the Terraform modules provisioning the supporting AWS infrastructure.
• Built the Prometheus/Grafana monitoring layer for job and pipeline health.

Producers publish events to Kafka. A Flink job consumes each topic, applies schema validation and stateful transformations (windowing, deduplication, enrichment), and checkpoints state to durable storage on a fixed interval. Validated, transformed records are written to a Hudi table on S3, partitioned for efficient querying. Job and pipeline metrics are scraped by Prometheus and visualized in Grafana.

• Used Flink's keyed state and RocksDB state backend to support stateful, exactly-once processing at scale.
• Enabled checkpointing with a durable, externalized checkpoint store so jobs can recover after a restart without reprocessing from the beginning of a topic.
• Applied schema validation at the ingestion boundary so malformed events are routed to a dead-letter path instead of breaking the job.
• Chose Apache Hudi for the lake layer to get upsert support and incremental querying rather than append-only Parquet files.

• Use Flink instead of a simpler consumer-based service.

  Flink's native support for event-time processing, windowing, and checkpointed state made it a better fit than hand-rolled consumer logic once the pipeline needed exactly-once guarantees.

• Use Hudi rather than plain Parquet for the lake tables.

  Streaming data frequently needs upserts and late-arriving corrections. Hudi's table format supports that without rewriting entire partitions.

• Checkpoint failures under state backend pressure.

  Tuned checkpoint interval and state backend configuration, and added checkpoint-duration and failure-count metrics to Grafana so regressions are visible before they cause job restarts.

• Duplicate records after task-manager restarts.

  Verified end-to-end exactly-once configuration (Kafka transactional producer settings plus Flink checkpointing) and added a deduplication step keyed on event identifiers as a defense-in-depth measure.

The resulting pipeline processes Kafka events through Flink with checkpointed, stateful transformations and lands validated, deduplicated records in a Hudi-managed lake table — recoverable from failure and observable end-to-end.

• Unit tests for transformation and schema-validation logic.
• Local integration tests using Docker Compose to run Kafka, Flink, and a MinIO-backed S3 substitute.
• Manual failure-injection testing (killing task managers mid-run) to verify checkpoint recovery behavior.

• Flink job metrics (checkpoint duration, backpressure, records processed) exported to Prometheus.
• Grafana dashboards for pipeline throughput, consumer lag, and checkpoint health.
• Structured logging for schema-validation failures routed to a dead-letter topic.

• Least-privilege IAM roles scoped to the specific S3 prefixes the job reads from and writes to.
• Encryption at rest for lake storage and in transit for Kafka connections.
• Secrets (broker credentials, storage keys) sourced from a secrets manager rather than environment files.

Verified results have not been published for this project yet.

• Exactly-once semantics require care at every hop (producer, processor, and sink), not just at the stream-processing layer.
• Investing in dead-letter handling early made schema drift a non-event instead of a pipeline outage.

• Add automated schema-evolution testing against a schema registry.
• Evaluate Apache Iceberg as an alternative table format for broader query-engine compatibility.

• Apache Kafka
• Apache Flink
• Python
• AWS S3
• Apache Hudi
• Terraform
• Grafana
• Docker