This project creates a real time data streaming pipeline that ingests MovieLens dataset events from Kafka topics into Delta Lake tables using Apache Spark Structured Streaming on Databricks. There is an optimization notebook that performs a couple experiments with data optimization on spark and created a separate analytics notebook for exploratory data analysis.