Evaluation methodology¶

This page explains the steps of a run, the metrics and the limits of the results. Datasets describes the data, and Benchmark results shows the results.

Steps of a run¶

For each detector and each dataset, the benchmark does these steps once, with seed 0:

It loads the data with Data.
It keeps some of the points, as the list after these steps shows. Then it scales each feature to the range from 0 to 1, with the minimum and the maximum of these points.
It makes the detector with its default parameters and random_state=0. Some detectors also get the arguments that they require, as the list below shows.
It gives each point to fit_score_partial, which scores the point and then trains the detector on it.
It computes the metrics. The first 256 points are the warm-up, and their scores do not count, because the detector has little or no model when it scores them. A stream with fewer than 1,024 points has a warm-up of a quarter of its points.

The parts keep different points, and they use different metrics:

odds: The first 5,000 points of an ODDS dataset, after a shuffle with random_state=0. Many ODDS files put all the anomalies at the end, so the order of the file is not a fair stream. The metrics are AUROC and AUPR.
drift: Every k-th point of a stream, in time order. k is the smallest step that leaves at most 10,000 points, so the stream keeps all its drifts. The metrics are AUROC and AUPR.
univariate, nab and shingles: All the points of a NAB series, in time order. The shingles part turns each value into a shingle of the last 10 values, and the first 9 shingles repeat the first value. The metrics are the NAB score, and AUROC on the series with anomalies.
univariate on UCR: All the points of a UCR series with at most 20,000 points, in time order. The metrics are the solved series, and AUROC after the training part.

Step 2 uses the minimum and the maximum of the data. A live stream does not know them in advance. For a live stream, take them from historical data, or use known physical limits.

These detectors get more arguments in step 3:

HalfSpaceTrees and RSHash get the range from 0 to 1 for each feature.
Inqmad gets the number of features.
LocalOutlierProbability fits on the warm-up points.
HTM gets each point with its timestamp, the range from 0 to 1, the probationary period of NAB, and the seed 0. It does not run on the UCR series, which have no timestamps.
KNNCAD learns during the probationary period of NAB, as in NAB.
RelativeEntropy gets the range from 0 to 1.
SeasonalESD and SeasonalHybridESD get the period of the series and a window of 3 periods and one point. The R package of their paper needs more than 2 periods. With an odd window, the detectors give the same anomalies as the R package.

Metrics¶

AUROC: The probability that a random anomaly gets a higher score than a random normal point. A random detector gets 0.5, and a perfect detector gets 1. Evaluate a detector explains AUROC and AUPR in more detail.
AUPR: The average precision over all the score thresholds. A random detector gets the fraction of anomalies in the dataset, and a perfect detector gets 1.
NAB score: NAB marks a window of points around each anomaly, and its score counts each window as one event. The first alert in a window detects it, and an early alert counts more than a late one. A window without an alert is a miss. An alert outside the windows is a false alert, and a false alert soon after a window costs less than others. NAB uses one alert threshold for all the series of a detector, the threshold that gives the highest score. Then it scales the score: a detector without alerts gets 0, and a perfect detector gets 100. The score leaves out the probationary period of each series, which is its first 15% of points, at most 750.
Solved series: A detector solves a UCR series when its highest score after the training part is at most 100 points from the anomaly. The authors of the archive use this rule.
Alerts of a calibrator: For each run of the odds part, the benchmark gives all the scores to each calibrator. Then it counts the alerts after the warm-up.

NAB has three profiles. The standard profile weights a false alert at 0.11 of a miss. The profile “low false alerts” doubles the weight of a false alert, and the profile “low misses” doubles the weight of a miss.

One threshold for all the series needs scores with the same scale on each series. HTM, KNNCAD and RelativeEntropy give scores from 0 to 1, as the detectors of NAB do. The benchmark gives the scores of the other detectors to ConformalProbabilityCalibrator, whose window holds all the earlier scores of the series. With its default window of 300 scores, many points tie at the highest value.

The benchmark uses the scoring code of NAB. With it, KNNCAD gets NAB scores of 58.0, 43.4 and 64.8, the scores that the scoreboard of NAB lists for its KNN-CAD detector. RelativeEntropy gets higher scores than on the scoreboard, because its histogram follows the paper. HTM gets 64.2, 56.8 and 68.3. The scoreboard lists 64.6, 56.7 and 69.2 for the numentaTM detector of NAB, which uses the same kind of temporal memory, and 70.1, 63.1 and 74.3 for the numenta detector, which uses the backtracking temporal memory of NuPIC.

What the results do not show¶

Tuned parameters can give very different results. For example, KitNet trains on its first 1,100 points by default. 7 ODDS datasets have fewer points, so on them, KitNet never ends its training. Choose a detector names the parameters with the largest effect.
The points of the ODDS datasets are independent, so these streams have no drift and no seasonality. The streams with drift show how a detector follows changes over time, but they are only 8 streams. Their thinning also makes the changes faster in points.
The NAB scoreboard has no results for MAD, SAD and the ESD detectors. Their NAB scores depend on the calibration, which NAB does not use. The periods come from the data, because NAB gives no periods.
The UCR results come from the short series only. A longer series can be harder for a detector.
Each detector runs with one seed, and the shingles have one size. Another seed or another shingle size can give other results. A change that draws the same random numbers in another order has the same effect, and on a small dataset it can move the AUROC a lot. For example, pull request 397 made RobustRandomCutForest cut all its trees in one batch. The model stayed the same, but the cuts of the trees came in another order. Its AUROC on glass with seed 0 went from 0.715 to 0.638, but over seeds 0 to 19 its mean AUROC on glass is 0.675 before the change and 0.686 after it.
A run on another platform can give slightly different results for Inqmad, and for LODA on series with many equal scores.
On a series with many equal points, Inqmad gives these points scores that are equal up to rounding, so the rounding decides their order and the AUROC. For example, on nab/realAWSCloudwatch/ec2_disk_write_bytes_1ef3de.csv with shingles, 3,217 of the 4,474 scored points, 322 of them anomalies, are copies of the first shingle. A change in rounding moved its AUROC from 0.529 to 0.617, but with the scores rounded to 9 decimals, the AUROC is 0.530 in both cases.
LocalOutlierProbability gives a probability, and on a stream far from its reference points, most probabilities are within 1e-9 of 1. The rounding to 9 significant digits makes these scores equal, so their order no longer decides the AUROC. For example, on nab/realAWSCloudwatch/ec2_cpu_utilization_ac20cd.csv with shingles, 3,588 of the 3,776 scored points round to 1, and all 403 anomalies are among them. Before the rounding, differences below 1e-9 ordered these points and gave an AUROC of 0.775. With the rounding, the AUROC is 0.528.
On a series of integer counts, many points of a window of IncrementalLOF are copies, so many LOF values are exactly 1 in exact arithmetic. In floating point, they differ in the last bits, and before pull request 544, these differences decided their order. For example, on nab/realTweets/Twitter_volume_GOOG.csv, 13,584 of the 15,586 scored points, 1,145 of them anomalies, have an LOF of 1 within 1e-14. With the rounding to 9 significant digits, these points tie, and the AUROC went from 0.543 to 0.521. The rounding moved the AUROC of 5 other series by 0.005 to 0.019, but the mean AUROC on NAB only from 0.519 to 0.518.
ReferenceWindowModel runs a PyOD model, so its results depend on that model. IForestASD is one example of it.
The anomalies also enter the window of a calibrator. On a dataset with many anomalies, fewer normal points pass the threshold, so the false alert rate can be lower than expected. The window of ConformalProbabilityCalibrator holds 300 scores by default, so its highest value is 299/300, and a threshold of 0.999 never raises an alert.