Pipeline / Streaming Deployment

<aside> 💡

In Databricks, pipeline == streaming

</aside>

• Streaming deployment involves deploying machine learning models that can handle real-time or streaming data.
• Instead of processing data in batches, these models can continuously perform inference as data arrives, providing low-latency and continuous insights.
• It often involves setting up a system that continuously receives and processes incoming data, makes predictions in real-time.

Pipeline Deployment

Advantages

• Lower Latency
• Generate predictions and act sooner.
• Not suitable for applications that require near real-time predictions.
• Event-driven architecture; enable systems to adopt quickly

Disadvantages

• More costly than batch solution.
• More complicated to develop, maintain and monitor compared to batch solution.
• Less throughput compared to batch solution.
• Resource intensiveness; computational power to process data.

Pipeline Deployment Graph