Use ML foundations to select the appropriate algorithm for a given model scenario
- supervised or unsupervised
- classification vs regression vs time series
Identify methods to mitigate data imbalance in training data
Compare estimators and transformers
Develop a training pipeline
Use Hyperopt's fmin operation to tune a model's hyperparameters
Perform random or grid search or Bayesian search as a method for tuning hyperparameters.
Parallelize single node models for hyperparameter tuning
Describe the benefits and downsides of using cross-validation over a train-validation split.
Perform cross-validation as a part of model fitting.
Identify the number of models being trained in conjunction with a grid-search and cross-validation process.
Use common classification metrics: F1, Log Loss, ROC/AUC, etc
Use common regression metrics: RMSE, MAE, R-squared, etc.
Choose the most appropriate metric for a given scenario objective
- When to use RMSE, MAE, F1, ROC/AUC, etc
Identify the need to exponentiate log-transformed variables before calculating evaluation metrics or interpreting predictions
- Exponentiating log-transformed variables is a way to reverse the logarithmic transformation and return the variable to its original scale.
Assess the impact of model complexity and the bias variance tradeoff on model performance
- Bias
- Bias measures the error introduced by approximating a real-world problem (which may be complex) with a simpler model.
- High bias occurs when the model is too simple to capture the underlying patterns in the data, leading to underfitting.
- Example: A linear model trying to fit a quadratic relationship will have high bias because it cannot capture the curvature.
- Variance
- Variance measures how much the model's predictions fluctuate for different training datasets.
- High variance occurs when the model is too complex, capturing noise along with the signal, leading to overfitting.
- Example: A high-degree polynomial model perfectly fitting the training data but performing poorly on new data.
- To reduce bias: Use a more complex model or features that better represent the problem
- To reduce variance: Use regularization, gather more data, or simplify the model.