General recommendations for MLOps

Deploy code, not models

In most situations, Databricks recommends that during the ML development process, you promote code, rather than models, from one environment to the next. Moving project assets this way ensures that all code in the ML development process goes through the same code review and integration testing processes. It also ensures that the production version of the model is trained on production code.

In this pattern, the code to train models is developed in the development environment. The same code moves to staging and then production. The model is trained in each environment: initially in the development environment as part of model development, in staging (on a limited subset of data) as part of integration tests, and in the production environment (on the full production data) to produce the final model.

Advantages:

• In organizations where access to production data is restricted, this pattern allows the model to be trained on production data in the production environment.
• Automated model retraining is safer, since the training code is reviewed, tested, and approved for production.
• Supporting code follows the same pattern as model training code. Both go through integration tests in staging.

Disadvantages:

• The learning curve for data scientists to hand off code to collaborators can be steep. Predefined project templates and workflows are helpful.

Also in this pattern, data scientists must be able to review training results from the production environment, as they have the knowledge to identify and fix ML-specific issues.

If your situation requires that the model be trained in staging over the full production dataset, you can use a hybrid approach by deploying code to staging, training the model, and then deploying the model to production. This approach saves training costs in production but adds an extra operation cost in staging.

Deploy models

In this pattern, the model artifact is generated by training code in the development environment. The artifact is then tested in the staging environment before being deployed into production.

Consider this option when one or more of the following apply:

• Model training is very expensive or hard to reproduce.
• All work is done in a single Databricks workspace.
• You are not working with external repos or a CI/CD process.

Advantages:

• A simpler handoff for data scientists