Overfitting Data: A Beginner’s Guide

Overfitting is a type of machine learning behavior where the machine learning model is accurate for training data but cannot generalize new information. Understanding overfitting and how to prevent it is essential for data professionals to build accurate machine learning models to gain insights and make predictions. Explore what a statistical model is, why overfitting occurs, and how you can take steps to prevent overfitting in machine learning models.

Read more: How Does Machine Learning Work?

What is a statistical model?

This is a mathematical model developed based on statistical analysis. Statistical models represent the relationship between different variables to explain complex research questions, make predictions, identify patterns, and test hypotheses. Statistical models generally involve two types of variables for the statistical analysis:

• Dependent variables (response or outcome variables): Outcomes or results we're interested in predicting or explaining

• Independent variables (predictor variables or explanatory variables): Variables we believe influence or cause the dependent variable

When using a statistical model, you generally try to determine how your independent variables affect your dependent variables. You can choose many types of statistical models depending on your industry and area of interest. Statistical models generally fall into two categories: Supervised and unsupervised learning techniques. 

Supervised learning includes regression and classification, while unsupervised learning includes algorithms such as clustering or association. Choosing an appropriate statistical model is vital to helping professionals understand data, identify patterns and relationships among variables, and make data-driven decisions.

Read more: Understanding Different Types of Data

What is overfitting?

Overfitting happens when a statistical model cannot accurately generalize from the training data. This means your model may be very accurate with inputs close to your training data but have a high error rate for new data. For example, imagine you're showing a child pictures of flowers from your garden. These flowers all have perfect lighting and a clean background. The child gets good at identifying flowers under these specific conditions based on your pictures. 

However, if they encounter a flower in the wild with different lighting or a cluttered background, they might not recognize it as a flower because their learning was too focused on the specific details of your pictures, not the general characteristics of flowers.

This scenario is similar to what happens with a statistical model during overfitting. When building a model, you start with a training data set. The model learns from this data, just like the child learns from pictures of flowers. If the model fits the training data too closely, its results may appear like those of the child who can only recognize flowers like those in the pictures. The model might perform very well on that specific data but struggle to perform on data outside the training set.

Read more: 10 Machine Learning Algorithms to Know

What is underfitting?

Underfitting happens when a data model has a high error rate for both the training and new data. This generally occurs when the model is too simple, either needing more training time, fewer restrictions, or more guidelines for what to identify. 

Underfitting can happen as a result of trying to prevent overfitting. Because overfitting can occur from a model adhering too closely to training data, you may prevent this by giving fewer inputs during the training phase. However, if you restrict your inputs too much, your model may need more restrictions to distinguish between information accurately.

Read more: Overfitting vs. Underfitting: What’s the Difference?

Why does overfitting happen?

You can’t prevent overfitting 100 percent of the time, but by identifying several triggers of overfitting, you can greatly reduce the likelihood of it occurring. Overfitting can occur for several reasons, including the following.

1. Lack of training data 

If your data set is small, the training data might not represent all of the types of input data your model is supposed to recognize. 

2. Noisy data

Overfitting can also happen if your training data has a lot of extraneous information. When you have too much extra information, your model might begin to recognize this noise as features of the data. Training a model for too long on sample data can also lead to it recognizing noise as part of the input parameters instead of the general patterns.

3. Lack of regularization

Another cause of overfitting is lack of regularization. Regularization is a technique you can use to prevent overfitting by adding a penalty to the loss function. It helps prevent the model from learning overly complex patterns in the data and keeps the model simpler. 

A loss function measures how far off the predictions of the model are from the actual values in the training data. You want to minimize this loss. However, without any constraints, a complex model might become too tailored to the training data, even capturing the noise or outliers. 

By adding a penalty term to the loss function, we make it so the model aims to minimize its prediction error on the training data. If you do not apply enough regularization, the model may overfit. 

How to prevent overfitting

You can employ one or several of the following methods to reduce the likelihood of overfitting. By taking these steps from the beginning, you can avoid re-doing your models later.

• Data augmentation: You can create new synthetic training samples by modifying the existing data. For instance, in image data, you can rotate, flip, or crop images to create new samples. This creates “new” training data from existing information and helps improve the model’s ability to generalize.

• Ensembling: You can combine the predictions of multiple models to give a final prediction. Bagging and boosting are two ensemble methods that train different models sequentially or in parallel, respectively.

• Regularization: Regularization aims to counteract overfitting my reducing accuracy on training data while improving accuracy on new data. Regularization finds the most important variables influencing your results and weights them more heavily compared to less important features. 

• Pruning: You can remove unnecessary structures from a model to make it simpler. It can reduce the complexity of the model and remove unnecessary noise. 

• Early stopping: During the training of a machine learning model, you can assess model performance on a validation set. At a certain performance threshold, you can stop the model’s training process. Doing so helps to prevent the model from learning noise as part of the training data.

Read more: What Is Big Data Analytics? Definition, Benefits, and More

Next steps on Coursera

Overfitting is a common machine learning obstacle where the algorithm learns the training data too specifically and has trouble generalizing to other data. This may happen when you have too little or noisy training data, but you can take steps to reduce the likelihood of this happening with ensembling, regularization, and more. 

You can learn more about overfitting, data analytics, and machine learning models from exciting courses and Specializations from top universities on the Coursera platform. Consider completing the Big Data Specialization or the Google Data Analytics Professional Certificate to build structured knowledge and gain job-ready skills at your own pace.