Stop removing outliers just because!

Learn how to tell your model to pay attention to outliers

STOP

Outliers are data points that stand out for being different from the remaining data distribution. An outlier can be:

  • An odd value in a feature
  • A data point distant from the centroid of the data
  • A data point in a region of low density, but between areas of high density.

Suppose you have been working in data science. In that case, you are already familiar with the concept, and you have probably integrated different methods in your pipelines to detect, transform, or even remove outliers from your data.

If so, be careful! Outliers do not respect the data distribution, so you should not pretend they do by removing inconvenient data points or transforming their features to become closer to the remaining data distribution. I know you need to handle them to avoid nonsensical values disturbing your pipeline. Still, you can also let the model know those data points are outliers instead of ignoring that information. The two main questions here are: Why? and How?

Why? Because you never know the cause of an outlier. It can represent either an error in the data acquisition process or a real anomaly in your population. This is highly relevant when dealing with Fraud Detection, Predictive Maintenance, or Compliance Validation situations. In these use cases, you want to detect the odd values (outliers) to prevent further risks.

How? Creating new features that represent how odd a data point is. If you make this information clear to the model, it will be able to detect the outliers by itself. But if outliers can be of different forms (as seen in the figure above), what features can represent their oddness? Find the answer in the next section!

 

Features for Outliers

Mahalanobis Distance

You can use different distance algorithms to compute how far a data point is from the centroid. We recommend the Mahalanobis distance since it is better to deal with multivariate outliers resulting from unusual combinations between multiple variables. For example, consider these three variables: weight, height, and gender. A height of 150 cm is not that unusual for the Portuguese female population, and a weight of 90kg is not uncommon for the Portuguese male. However, a female Portuguese with 150 cm and 90 kg would be very unique.

Density Estimation

 

Previously we saw a feature that measures the distance to the centroid, but what if the data distribution has a shape like the one in the figure below?

In this case, an outlier can be a data point located in the regions where the density distribution has a depression, no matter the distance to the centroid. So a good feature to represent it would be the data density in the neighborhood of the data point.

You can use methods like KDE (Kernel Density Estimation) to estimate the density. However, this method can be too computationally expensive. So we propose a more straightforward and cheaper method: binning. 

There are two ways of using binning to estimate density distribution:

  • Use bins with equal widths: Split the data into equal-width bins and compute the density of each bin. The fewer points the bin has, the more normal the data point is.
  • Use bins with equal frequencies: Split the data into equal-frequency bins and compute the width of the bin. The larger the bin, the more abnormal the data point is

 

Autoencoders Reconstruction

Training an autoencoder with your data will let the encoder learn the data distribution of the different variables and their relationship. Then, when the autoencoder receives a data point that deviates from the remaining data, it won’t be able to reconstruct the data point correctly.

A good feature to represent outliers would be the distance between the input, X, and the output, X’ (e.g., cosine distance). Higher distances will be correlated with odder data points.

Do you want to further discuss this idea?

Book a meeting with Francisca Morgado

Meet Francisca Learn More

Final Remarks

Now that you know how to detect outliers, you have a new trick to detect possible frauds, anomalies, or errors without needing to collect data for all those exceptions. Here, we presented you with three different ways to do so.

For more ideas on how to get the most out of your data, subscribe to our newsletter below and stay tuned.

Like this story?

Subscribe to Our Newsletter

Special offers, latest news and quality content in your inbox once per month.

Signup single post

Consent(Required)
This field is for validation purposes and should be left unchanged.

Recommended Articles

Article
How Often Should You Retrain Machine Learning Models?

A common question in the domain of AI and machine learning is: how often should you retrain machine learning models? The answer isn’t as straightforward as you might think. It’s not a one-size-fits-all solution, but rather a process that requires careful consideration and strategic planning. In this article, we’ll explore three strategies for deciding when […]

Read More
Article
Reject Option: Your AI Model Has the Right to Remain Silent

When it comes to AI models, we often expect them always to provide an answer. But what if we could trust them more when they choose to remain silent? This concept, known as the ‘Reject option‘, allows AI models to abstain from answering when they are not confident, opening up many applications in your business […]

Read More
Kelwin Fernandes at the leacture the 3 tribes of AI. UPdate2024
Article
Empowering Future Leaders with AI: NILG.AI at UPdate 2024

At NILG.AI, we were recently proud sponsors at UPdate 2024. We lead both a workshop and a lecture by our CEO, Kelwin Fernandes, who has over 15 years of experience in AI. Our goal? To empower future leaders and professionals to navigate the complex yet rewarding terrain of AI with confidence and safety. UPdate 2024, […]

Read More