Outlier Detection: Finding What Doesn’t Fit

A fraudulent payment, a failing sensor, a typo that turns 32 into 320: the data points that matter most are often the ones that don’t look like the rest. Outlier detection finds them, usually without any examples of what “wrong” looks like, by learning what normal looks like and flagging what doesn’t fit. The trick is that every method has its own idea of normal, and outliers can disguise themselves by bending it. Here you’ll watch late deliveries hide from a textbook rule, then compare four methods on data with more than one kind of normal.

Rules for one number#

Start with something simple: how many minutes each delivery took. Three classic rules decide what’s unusual.

• z-score: how many standard deviations a value is from the mean. Beyond 3 is the usual cut-off for an outlier.
• IQR fences: John Tukey’s rule from 1977. Take the middle half of the data, between the first and third quartiles, and flag anything more than 1.5 times that range beyond it.
• Modified z-score: like the z-score, but measured from the median, with the median absolute deviation (MAD) in place of the standard deviation. Boris Iglewicz and David Hoaglin suggested flagging values above 3.5.

The difference is what each rule leans on. The mean and the standard deviation are pulled around by every value, including the outliers. The median and the quartiles hardly notice a few extreme values: they’re robust.

One 80-minute order is caught by all three rules. Add four and the z-score rule catches none of them: the mean rises to 36.6 minutes, the standard deviation balloons from 4.4 to 14.5, and the cut-off moves past 80. The outliers have hidden each other, an effect statisticians call masking. Even three orders of 55 minutes are enough to hide from it. The IQR fences and the modified z-score catch every one, because the quartiles and the median barely move.

Rules built on the mean and standard deviation are bent by the very points they’re meant to find. Prefer the median-based rules, or fit the mean and spread on data you’ve already cleaned. And beware tiny samples: with 10 values or fewer, no value can ever be more than 3 standard deviations from the mean, however extreme it is. (The largest possible z-score is (n − 1)/√n, which is 2.85 for 10 values.)

When normal has more than one shape#

Real data has many columns, and normal is rarely one tidy blob. Customers come in groups, machines have several modes, and a value that’s fine in one group is strange in another. Here are four ways to score how unusual each point is:

• Distance from the centre: the multi-column version of the z-score (the Mahalanobis distance), which allows for how spread and tilted the whole cloud is.
• Nearest neighbours: how far a point is from its 10th-nearest neighbour. Points in empty space score high.
• Local outlier factor (LOF): from Markus Breunig and colleagues in 2000. It compares how crowded a point’s neighbourhood is with how crowded its neighbours’ neighbourhoods are, so it judges each point against its own surroundings.
• Isolation forest: from Fei Tony Liu, Kai Ming Ting and Zhi-Hua Zhou in 2008. It makes random cuts through the data and measures how few cuts it takes to get a point on its own. Odd points are easy to isolate.

Flagging the 10 highest scores, distance from the centre catches only B; the other nine flags go to ordinary points on the edge of the loose group, and A ranks 32nd. Nearest-neighbour distance catches B and C but ranks A 27th. The local outlier factor puts A 2nd, B 1st and C 9th. The isolation forest catches all three as well (5th, 1st and 9th), though with other random trees A lands anywhere from 3rd to 19th, in the top ten about three times in four.

Look at the pink areas, where a new point would be flagged. Distance from the centre draws one big oval and ignores the groups entirely. Nearest neighbours draws a similar margin round both groups: anything with ten points within reach looks normal, so A, a short step from a crowd, passes. LOF wraps each group at its own scale, which is why it also flags a few perfectly ordinary points on the edge of the tight group. The isolation forest’s straight cuts give boxy edges. No method is best everywhere: each one encodes a different idea of what normal looks like.

Outliers in practice#

• You choose how many to flag. Every method gives a score; turning scores into flags means picking a cut-off, often as an expected share of outliers (scikit-learn calls it contamination). That’s a business decision: how many alerts can someone actually check?
• Scale your columns first. Distance-based methods treat one unit of each column as equal, so a column measured in pounds swamps one measured in thousands of pounds.
• Flagged isn’t the same as wrong. An outlier might be a typo, a fraud, or your best customer. Fix or drop data errors; keep real extreme values, and look at them, because they’re often the most interesting rows you have.
• Check against known cases. If you have even a few confirmed frauds or faults, see where they rank. A detector that ranks them in the middle isn’t finding what you care about.
• For images, text and sensor streams, an autoencoder can learn what normal looks like and flag what it can’t rebuild.

In code#

Both of the figure’s best performers are in scikit-learn (pip install scikit-learn). This builds data shaped like the figure’s, from a fresh random draw:

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor

rng = np.random.default_rng(0)
tight = rng.normal([-1.6, 0.4], 0.28, size=(100, 2))
loose = rng.normal([1.5, -0.3], 0.9, size=(100, 2))
planted = np.array([[-1.6, 1.55], [4.2, 2.9], [-0.2, -1.6]])   # A, B, C
X = np.vstack([tight, loose, planted])

iso = IsolationForest(n_estimators=100, max_samples=128, random_state=0).fit(X)
lof = LocalOutlierFactor(n_neighbors=10).fit(X)

for name, score in [("isolation forest", -iso.score_samples(X)),
                    ("local outlier factor", -lof.negative_outlier_factor_)]:
    rank = (-score).argsort().argsort() + 1          # 1 = most unusual
    print(f"{name}: A #{rank[200]}, B #{rank[201]}, C #{rank[202]}")
# isolation forest: A #15, B #1, C #13
# local outlier factor: A #2, B #4, C #20

With different random points the ranks shift, as they did when you grew a new forest in the figure: here the isolation forest misses A and C in its top ten, while LOF still puts A near the top. Run any method on a few random draws, or with a few seeds, before trusting a ranking. To flag new data later, use IsolationForest.predict, or create LocalOutlierFactor(novelty=True).