What is AutoML

Automated machine learning (AutoML) is a fully-automated model development solution. AutoML technologies usually automate the ML process from data to model selection.

Main Points Solved

• Quickly verify the predictive power of a dataset
• Get a baseline Model to guide project direction

How does AutoML work?

1. Cleans and prepares your data.
2. Orchestrates distributed model training and hyperparameter tuning across multiple algorithms.
3. Finds the best model using open source evaluation algorithms from scikit-learn, xgboost, LightGBM, Prophet, and ARIMA.
4. Presents the results. AutoML also generates source code notebooks for each trial, allowing you to review, reproduce, and modify the code as needed.

Code

from databricks import automl
summary = automl.regress(train_pdf, target_col="price", timeout_minutes=10)

model_uri = summary.best_trial.model_path

# run inference using the best model
model = mlflow.pyfunc.load_model(model_uri)
predictions = model.predict(X_test)
test_pdf["MedHouseVal_predicted"] = predictions

# create udf
predict_udf = mlflow.pyfunc.spark_udf(spark, model_uri)
test_df.withColumn("price", predict_udf())