Real-time Deployment

• The process of deploying and serving machine learning models in a production environment where predictions are generated instantly in response to incoming data or request.
• Crucial for applications that require low-latency responses, such as fraud detection, autonomous systems, and other time-sensitive tasks.

With the emergence of new Gen AI applications, this deployment method is becoming increasingly common, especially as large language models need to be served in real-time.

Typical Real-time Deployment Use Case

• Description: Real-time Fraud Detection in Online Banking
• Scenario:
  • A pre-trained machine learning model, specifically designed for fraud detection, is deployed in the real-time environment.
  • The model provides immediate predictions over a REST API.
  • An immediate decision is made to flag the transaction.
• Requirements: low latency, immediate action, 24/7 uptime, continuous monitoring

Challenges with building Real-time ML Systems

<aside> 💡

Most ML models don’t get into production

</aside>

• ML infrastructure is hard
  • Real-time ML systems require fast and scalable serving infrastructure, which is costly to build and maintain.
• Deploying real time models needs disparate tools
  • Data teams use diverse tools to develop models
  • Customers use separate platforms for data, ML and Serving, adding complexity and cost
• Operating production ML requires expert resources
  • Steep learning curve of deployment tools.
  • Model deployment is bottlenecked by limited engineering resources, limiting the ability to scale AI

Databricks Model Serving