Databricks Modle Serving

<aside> 💡

Unified UI, API & SDK for managing all types of AI Models

</aside>

Foundation and External Models are covered in the GEN AI Course.

Core Features of Model Serving

<aside> 💡

Support real-time production ML workloads

</aside>

• Real Time
  • Low overhead latency: <100ms
  • Throughput: 3K+ QPS
  • Availability: 99.5%
  • Scalable: Automatically scales up/down to handle bursty traffic
  • Secure: PrivateLink and IP-allowlist
• Lakehouse Unified
  • Feature Store Integrated: Automated feature/vector lookups
  • MLflow Integrated: Fast & easy model deployment
  • Quality & Diagnostics: Built-in-metrics, integrate with Lakehouse Monitoring
  • Unified governance: Manage data & AI with UC
• Simplified Deployment
  • Simple: Endpoints UI and API for simple deployment
  • CPU and GPU Support
  • Flexible: Traffic splitting for staged roll-out and A/B testing

Feature: Serverless Compute

<aside> 💡

Quickly scale up/down to handle your bursty traffic

</aside>

• GPU Support: Deploy LLMs with ease
• Fast Autoscaling: Compute is managed and kept warm by Databricks, allowing rapid scaling.
• Scale within Workload size: Autoscale within selected range.
• Scale to zero: Save costs for use cases with predictable, non-24/7 traffic

Feature: MLflow Integration

• Faster path to production: Natively connects to MLflow Model Registry, enabling fast and easy deployment of models.