MLflow

• MLflow is a versatile, expandable, open-source platform for managing workflows and artifacts across the ML lifecycle
• Built-in integrations with many popular ML libraries, but can be used with any library, algorithm, or deployment tool
• MLflow Tracking: An API for logging parameters, code versions, metrics, model environment dependencies, and model artifacts when running your ML code. MLflow Tracking has a UI for reviewing and comparing runs and their results
• MLflow Models: A model packaging format and suite of tools that let you easily deploy a trained model (from any ML library for batch or real-time inference on platforms such as Docker, Apache Spark, Databricks, Azure ML and AWS SageMaker
• MLflow Model Registry: A centralized model store, set of APIs, and UI focused on the approval, quality assurance, and deployment of an MLflow Model.
• MLflow Projects: A standard format for packaging reuseable data science code that can be run with different parmeters to train models, visualize data, or perform any other data science task.
• MLflow Recipes: Predefined templates fordeveloping high-quality models for a variety of common tasks, including classifcation and regression.

MLflow is used by data scientists and MLOps professionals. MLOps use MLflow to securely deploy models to production and monitor them.

MLflow Use Cases

• A team of data scientists uses MLflow Tracking to record parameters and metrics from their experiments on a single problem domain. They use the MLflow UI to compare results and guide their exploration of the solution space. They store the outputs of their runs as MLflow models.
• An MLOps engineer uses the MLflow UI to compare the performance of different models and selects the best one for deployment. They register the model in the MLflow Registry to track this specific version’s performance in production.
• An MLOps engineer uses MLflow models to deploy a model to a production environment. They use MLflow Registry to track the model’s performance and to compare it to other models in production.
• A data scientist beginning work with a new project structures their code as an MLflow Project so that they can easily share it with others and run it with different parameters.

Unity Catalog setup

import mlflow

mlflow.set_registry_uri("databricks-uc")
CATALOG_NAME = "main"
SCHEMA_NAME = "default"

Read Data

import pandas as pd

white_wine = pd.read_csv("/databricks-datasets/wine-quality/winequality-white.csv", sep=";")
red_wine = pd.read_csv("/databricks-datasets/wine-quality/winequality-red.csv", sep=";")

red_wine['is_red'] = 1
white_wine['is_red'] = 0

data = pd.concat([red_wine, white_wine], axis=0)

# Remove spaces from column names
data.rename(columns=lambda x: x.replace(' ', '_'), inplace=True)