What is Hyperparameter

<aside> 💡

Definitions

</aside>

• Hyperparameter is a parameter whose value is used to control the training process.
• Hyperparameter Tuning (model tuning) is the process of adjusting the hyperparameters of a model to find the best configuration that maximizes its performance on a given dataset.

Examples

• Random Forest
  • Maximum depth of trees
  • Number of trees
  • Number of features
• Linear Regression
  • Normalize features or not
• Neural Networks
  • Activation function
  • Number of in each hidden layer

Hyperparameter Search Methods

<aside> 💡

How to pick a hyperparameter

</aside>

Grid Search

<aside> 💡

Systematic parameter selection

</aside>

• Define a grid of hyperparameter values and exhaustively searching through all possible combinations
• Evaluate each combination using cross-validation and selects the best set of hp.
• Can be computationally expensive

Random Search

• Randomly samples hyperparameter (hp) values from specified distributions
• Random search is more efficient than grid search, especially when dealing with large datasets or hp’s