Data¶

class pysad.utils.Data(data_base_path=None)[source]¶

A helper class to load and stream anomaly detection datasets.

Datasets are loaded from the local filesystem or downloaded on demand. The multivariate datasets come from the Outlier Detection DataSets (ODDS) repository mirror ([Ray16], hosted in the PyOD repository at a fixed commit). The univariate series come from the Numenta Anomaly Benchmark (NAB) and the UCR Time Series Anomaly Archive. The multivariate streams in time order come from KDD Cup 99, the Server Machine Dataset (SMD) and INSECTS. Each download must match a SHA-256 checksum. datasets() lists the datasets, and get_stream() gives the stream of a run of the PySAD benchmark.

Downloaded datasets are cached in a local directory. The cache directory defaults to ~/.cache/pysad, which can be overridden via the PYSAD_DATA_DIR environment variable or by passing an explicit data_base_path.

Parameters:: data_base_path (str or None) – Base directory that contains the data files and caches the downloaded datasets. If None, the class uses the directory in PYSAD_DATA_DIR if set, or ~/.cache/pysad (Default=None).

Methods

`datasets`([source])	Returns the names of the datasets that this class can download.
`describe`(data_file[, download])	Returns the number of points, the number of features and the anomaly rate of a dataset.
`get_data`(data_file[, download])	Loads the data given the filename or path, downloading it if not present.
`get_iterator`(data_file[, shuffle, ...])	Returns an iterator over the instances and labels of a dataset, in shuffled order by default.
`get_stream`(data_file[, random_state, ...])	Returns a shuffled dataset as arrays, in the order of `get_iterator()`.

static datasets(source='odds')[source]¶

Returns the names of the datasets that this class can download.

Parameters:

source (str) –

The collection of the datasets (Default=”odds”). These are the collections:

"odds": the 17 multivariate datasets of ODDS, for example "cardio.mat".
"nab": the 58 univariate series of the Numenta Anomaly Benchmark, for example "nab/realKnownCause/nyc_taxi.csv". The points in an anomaly window of NAB have the label 1.
"ucr": the 250 univariate series of the UCR Time Series Anomaly Archive, for example "ucr/025_UCR_Anomaly_DISTORTEDInternalBleeding14_2800_5607_5634.txt". The last two numbers of a name are the first and the last point of the anomaly, counted from 1. These points have the label 1. The number before them is the length of the training part, which has no anomalies. The first series download the archive of 184 MB.
"kddcup99": "kddcup99/http.csv", the 58,724 http connections of logged in users of the KDD Cup 99 10% file, in their original order. The 3 features are log(x + 0.1) of the duration, the bytes from the source and the bytes to the destination. The attacks have the label 1. The first use downloads the file of 74 MB.
"smd": the test halves of the 28 machines of the Server Machine Dataset, for example "smd/machine-1-1.txt", with one point of 38 features each minute.
"insects": the 4 INSECTS streams of the ARCUS paper, for example "insects/abrupt.csv", with 33 features. The southern house mosquito is the anomaly class.

Returns:

The names of the datasets, which get_data() accepts.

Return type:

list[str]

Raises:

ValueError – If source is not one of the collections above.

describe(data_file, download=True)[source]¶

Returns the number of points, the number of features and the anomaly rate of a dataset.

Parameters:

data_file (str) – Path or filename of data.
download (bool) – Whether to download the dataset if not present locally (Default=True).

Returns:

points (int), features (int) and anomaly_rate (float, the fraction of points with label 1).

Return type:

dict

get_data(data_file, download=True)[source]¶

Loads the data given the filename or path, downloading it if not present.

Parameters:

data_file (str) – Path or filename of the data.
download (bool) – Whether to download the dataset if not present locally (Default=True).

Returns:

The feature vectors of shape (num_instances, num_features) and the labels of shape (num_instances,).

Return type:

tuple[np.ndarray, np.ndarray]

get_iterator(data_file, shuffle=True, random_state=None, download=True)[source]¶

Returns an iterator over the instances and labels of a dataset, in shuffled order by default.

Parameters:

data_file (str) – Path or filename of data.
shuffle (bool) – Whether to shuffle the instances (Default=True).
random_state (int, np.random.RandomState or None) – Seed or random number generator for the shuffle, as in ArrayStreamer. None draws from NumPy’s global random state (Default=None).
download (bool) – Whether to download the dataset if not present locally (Default=True).

Returns:

Iterator over (X, y) pairs, where X is the feature vector of shape (num_features,) and y is its label.

Return type:

Iterator[tuple[np.ndarray, Any]]

get_stream(data_file, random_state=None, max_points=None, scale=False, download=True)[source]¶

Returns a shuffled dataset as arrays, in the order of get_iterator().

With random_state=seed, max_points=5000 and scale=True, this is the stream of the PySAD benchmark for that seed.

Parameters:

data_file (str) – Path or filename of data.
random_state (int, np.random.RandomState or None) – Seed or random number generator for the shuffle. None draws from NumPy’s global random state (Default=None).
max_points (int or None) – The number of first points of the shuffled stream to keep. None keeps all of them (Default=None).
scale (bool) – Whether to scale each feature to the range from 0 to 1, with the minimum and the maximum of the points that it keeps. A constant feature becomes 0 (Default=False).
download (bool) – Whether to download the dataset if not present locally (Default=True).

Returns:

The feature vectors of shape (num_points, num_features) and the labels of shape (num_points,), in stream order.

Return type:

tuple[np.ndarray, np.ndarray]