Concepts

This page explains the ideas behind PySAD. It shows how a streaming detector differs from a batch detector, what each component does, and why PySAD measures detectors as it does. For step-by-step instructions, see Getting started and the how-to guides.

Streaming anomaly detection

A stream is data that arrives one point at a time and does not end. A point is one record of the data, for example one sensor reading. An anomaly is a point that does not fit the pattern of the other points. Examples are a spike in a sensor reading, a fraudulent payment or a failing server.

A batch detector, such as the detectors in PyOD, trains on a complete dataset and then scores it. A stream has no complete dataset. A stream sets three constraints:

• Points arrive one at a time, and the detector must score each point before the next one arrives.

• The stream does not end, so the detector cannot keep every point in memory.

• The data drifts: normal behavior changes over time. The detector must follow the change.

A streaming detector learns from each point once and keeps a summary of the stream instead of the full stream. The summary is often a window: the most recent points or scores, up to a fixed number. The detector updates its model of normal behavior with each new point. It gives each point an anomaly score: a number that is higher for a more anomalous point.

The pipeline

A PySAD pipeline is a loop over the stream. Each point goes through the same stages:

Only the stream and the detector are necessary. Each stage is an object with its own state, so you can add, remove or replace a stage without a change to the other stages.

Stage	Module	What it does
Stream	pysad.utils	Gives the points one at a time. Your own loop over a file, a socket or a queue also works.
Preprocess	pysad.transform.preprocessing, pysad.transform.projection	Scales a point, removes seasonality or reduces the number of features.
Detect	pysad.models	Learns from the point and gives it an anomaly score.
Postprocess	pysad.transform.postprocessing	Smooths or normalizes the scores over a window.
Ensemble	pysad.transform.ensemble	Combines the scores of several detectors into one score.
Calibrate	pysad.transform.probability_calibration	Converts a score into a value between 0 and 1 with the same meaning for every detector.
Evaluate	pysad.evaluation	Measures the scores against labels (anomaly or normal), one point at a time.

The model interface

Every detector extends BaseModel and has two single-point methods. The API reference calls a point an instance:

fit_partial(x)

Updates the model with the point x, a NumPy array of shape (num_features,). A detector that keeps points keeps its own copies, so you can reuse one array for every point.

score_partial(x)

Gives x an anomaly score, a Python float. This method does not change the model, and it neither changes nor keeps x.

fit_score_partial(x) scores x with the model from the earlier points, and then learns from x. This order is called prequential evaluation, or test-then-train. Stream libraries and most detector papers use it, and a detector never scores a point that it learned from. Before a detector learns from any point, it gives a fixed score, usually 0. SeasonalESD and SeasonalHybridESD are the exceptions: as in the paper, they test the window that holds the newest value. Most loops use fit_score_partial.

BaseModel builds the batch methods fit(X), score(X) and fit_score(X) from the single-point methods. They take an array of shape (num_instances, num_features), one row per point, and process its rows in order, like a stream. Batch scores are a NumPy float64 array of shape (num_instances,).

Preprocessors, projectors, postprocessors, ensemblers and calibrators follow the same pattern with fit_partial, transform_partial and fit_transform_partial. Running statistics, for example a running mean, use update and get. Most of them also have remove.

Because score_partial does not change the model, you can score a point without learning from it. For example, you can stop calls to fit_partial after a training period and only call score_partial after it.

Higher means more anomalous

Every output in PySAD has the same direction: a higher value means a more anomalous point. This rule applies to detectors, postprocessors, ensemblers and calibrators. A threshold is the value at or above which you raise an alert. You can replace one component with another, and the same threshold logic still works.

The scale of the raw scores is different for each detector. Some detectors give a negative log density, which is high where points are rare. Some give a count and some give 0 or 1. Do not compare raw scores of two different detectors. Calibrate the scores first.

Calibration

A calibrator converts a raw score into a value between 0 and 1. PySAD has four calibrators:

AnomalyLikelihoodProbabilityCalibrator

Gives the anomaly likelihood of HTM (Hierarchical Temporal Memory), the detector of Numenta. It fits a Gaussian to the averages of each 10 consecutive recent scores. It returns one minus the probability that such an average is further from the mean than the average of the last 10 scores, on the same side of the mean. The value is 0.5 at the mean and close to 1 far from it. It expects scores between 0 and 1.

ConformalProbabilityCalibrator

Gives the fraction of recent scores that are lower than the new score. A value of 0.99 means that the new score is higher than 99% of the scores in the window. In statistics, this value is one minus the conformal p-value. It makes no assumption about the distribution of the scores.

GaussianTailProbabilityCalibrator

Fits a Gaussian (bell-shaped) distribution to the mean and variance of recent scores. It gives the probability that a value from this distribution is lower than the new score. It assumes that the scores are approximately Gaussian.

SPOTProbabilityCalibrator

Fits a Generalized Pareto distribution to the highest scores with extreme value theory. It gives one minus the probability that a score is higher than the new score. It makes no assumption about the distribution of the scores, and it estimates the probability of scores that are higher than all scores seen so far.

With the conformal calibrator, a threshold such as 0.99 has the same meaning for every detector. If most recent scores come from normal points, about 1% of normal points raise an alert. A false alert is an alert for a normal point. To get fewer false alerts, increase the threshold.

Prequential evaluation

Prequential evaluation (predict, then train) is the standard way to measure a detector on a stream. A metric is a measure of how well the scores find the anomalies. In prequential evaluation, the detector scores each point first, the metric records the score, and then the detector learns from the point. Each score comes from a model that did not see the point before, so the measurement shows how the detector does on new data.

for x, label in streamer.iter(X, y):
    score = model.score_partial(x)
    model.fit_partial(x)
    metric.update(label, score)

Most examples in this documentation call fit_score_partial instead. It scores each point before the detector learns from it, as the model interface explains. It also matches a live system, where the detector learns from every point that it scores. evaluate_stream() runs this loop for a labeled stream and returns the value of each metric.

Detectors need some data before their scores are stable. This first part of the stream is the warm-up period. A metric over the full stream includes the warm-up period. To see how the results change over time, compute the metric separately on consecutive parts of the stream, as Evaluate a detector shows.

Memory and time

Most PySAD detectors keep a summary of the stream, not the stream itself. The summary is one of these:

• A window of recent points, for example in ExactStorm, whose window slides by one point, and xStream, which replaces its window every window_size points.

• Histograms or sketches (compact counts of where points fall), for example in LODA and RSHash.

• A set of trees, for example in HalfSpaceTrees and RobustRandomCutForest.

• Running statistics, for example in StandardAbsoluteDeviation.

For these detectors, the memory has a fixed upper limit, and the time to process one point does not increase as the stream gets longer. A few detectors keep data that grows with the stream. Detectors shows the memory of each detector, so that you can choose one for a stream with no end.

Drift and forgetting

Normal behavior changes over time. This change is drift. A detector follows drift only if it forgets old data. PySAD detectors handle old data in three ways:

How it forgets	Detectors	Effect
A window of recent data	xStream, HalfSpaceTrees, ExactStorm, ApproxStorm, MCOD, NETS, CPOD, D3, ThreshLEAP, IncrementalLOF, KELOS, IForestASD, SeasonalESD, SeasonalHybridESD, KNNCAD, MatrixProfile, DAMP, MedianAbsoluteDeviation, SpectralResidual	Data older than the window has no effect. A smaller window follows drift faster.
Time decay	RSHash, RobustRandomCutForest, Inqmad, OnlineOneClassSVM, ChangeFinder, AnyOut, HoltWinters	Old points count less and less. A larger decay (time_decay for RobustRandomCutForest, alpha for Inqmad, learning_rate for OnlineOneClassSVM, discount for ChangeFinder, alpha, beta and gamma for HoltWinters) follows drift faster. Inqmad learns only from points that it classifies as normal. With the default time_decay=0, RobustRandomCutForest keeps a uniform sample of the whole stream and does not forget.
No forgetting	LODA, ADeMS, StandardAbsoluteDeviation, RelativeEntropy, TEDA	Every point since the start counts equally, so the detector reacts more slowly to drift as the stream grows.

DILOF keeps the newest half of its window. Each time the window is full, it replaces the oldest half with a summary half its size, so old points count less and less.

KitNet learns only during its grace periods, the first points of the stream. LocalOutlierProbability learns only from its initial points. After that, these two detectors do not follow drift.

A detector that does not forget can still follow a large, permanent change if you start a new detector after the change. Run a detector in production shows how.

Reproducible results

Some detectors use random numbers, for example to draw projections or tree splits. Each of them takes a random_state parameter. An int gives the same result on every run. None, the default, draws from the global NumPy random state.

To make a complete script reproducible, pass random_state to each component, or call pysad.utils.fix_seed() once at the start. fix_seed seeds the global state that random_state=None uses.

Labels

A label tells whether a point is an anomaly or normal. Most detectors are unsupervised: they learn without labels. The y parameter of fit_partial exists so that every detector has the same interface. Unsupervised detectors ignore it. Metrics use labels to measure a detector. The detector never sees the labels.