Anomaly Detection in R: Identifying Point and Collective Anomalies, Lecture notes of Statistics

Download Anomaly Detection in R: Identifying Point and Collective Anomalies and more Lecture notes Statistics in PDF only on Docsity!

What is an anomaly?

ANOMALY DETECTION IN R

Alastair Rushworth

Data Scientist

Defining the term anomaly

Anomaly: a data point or collection of data points that do

not follow the same pattern or have the same structure as

the rest of the data

Visualizing point anomalies with a boxplot

boxplot(temperature, ylab = "Celsius")

Collective anomaly

An anomalous collection of data instances

Unusual when considered together

Example: 10 consecutive high daily temperatures

Testing the extremes with

Grubbs' test

ANOMALY DETECTION IN R

Alastair Rushworth

Data Scientist

Visual assessment is not always reliable!

boxplot(temperature, ylab = "Celsius")

Checking normality with a histogram

Symmetrical & bell shaped?

hist(temperature, breaks = 6)

Running Grubbs' test

Use the grubbs.test() function:

grubbs.test(temperature)

Grubbs test for one outlier data: temp G = 3.07610, U = 0.41065, p-value = 0. alternative hypothesis: highest value 30 is an outlier

Get the row index of an outlier

Location of the maximum

Location of the minimum

which.max(weights)

[1] 5

which.min(temperature)

[1] 12

Let's practice!

ANOMALY DETECTION IN R

Monthly revenue data

Grubbs' test not appropriate here

Seasonality may be present

May be multiple anomalies

head(msales)

sales month 1 6.068 1 2 5.966 2 3 6.133 3 4 6.230 4 5 6.407 5 6 6.433 6

Visualizing monthly revenue

plot(sales ~ month, data = msales, type = 'o')

Seasonal-Hybrid ESD algorithm output

sales_ad <- AnomalyDetectionVec(x = msales$sales, period = 12, direction = 'both')

sales_ad$anoms

index anoms 1 14 1. 2 108 2.

Seasonal-Hybrid ESD algorithm plot

AnomalyDetectionVec(x = msales$sales, period = 12, direction = 'both', plot = T)