How to mitigate overfitting?

Related Questions:
– What is Overfitting?
– What is Underfitting?
– What is the Bias-Variance Tradeoff?

Overfitting is a common problem in machine learning where a model fits the training data too closely, resulting in poor performance on new or unseen data. To mitigate overfitting, there are several strategies that can be employed:

1. Use more training data: Increasing the amount of training data can help the model better capture the underlying patterns in the data and reduce overfitting. However, if the additional data is not providing additional information to the model, more data alone will not necessarily improve performance. Therefore, it is important to get data that provides more information to the model, for example by getting more samples closer to the decision boundary.
   
2. Use data augmentation: Data augmentation techniques can artificially increase the size of the training dataset by generating new examples with variations of the original data. This can help the model generalize better to new data.
   
3. Reduce the complexity of the model: If the model is too complex, it may be fitting the noise in the data instead of the underlying patterns. Therefore, reduce the complexity of the model, such as by reducing the number of layers or neurons in case of neural networks, and reducing the order of polynomial in case of regression etc.
   
4. Use regularization techniques: Regularization techniques, such as L1 or L2 regularization, can help prevent overfitting by adding a penalty term to the loss function that discourages large weights or complex models.
   
5. Use dropout: Dropout is another regularization technique used for Neural Networks where random neurons are temporarily removed during training, forcing the model to learn more robust and generalizable features.
   
6. Reduce feature set: Reduce the number of features either by using feature selection methods or using dimensionality reduction.
   
7. Early stopping: For models that use an iterative learning procedure (for ex: Gradient Descent) Early stopping can help with overfitting. Early stopping involves monitoring the model’s performance on a validation dataset during training and stopping the training process when the validation error stops improving. This can help prevent overfitting by stopping the model before it starts to fit the noise in the data.
   
8. Use cross-validation: Cross-validation can help estimate the performance of the model on unseen data by training and evaluating the model on multiple subsets of the data.

It is important to note that mitigating overfitting is a balancing act. Reducing the complexity of the model or increasing the regularization strength can also lead to underfitting. Therefore, it is important to carefully monitor the model’s performance on both the training and validation data to ensure that it is not overfitting or underfitting. 

Video Explanation

In this video, Andrew Ng explains some of the key methods used for mitigating overfitting including getting more data, feature selection and regularization (Runtime: 8:15 mins)