pysad.evaluation.WindowedMetric

class pysad.evaluation.WindowedMetric(metric_cls, window_size, ignore_nonempty_last=True, **kwargs)

A helper class to evaluate windowed metrics. The distributions of the streaming model scores often change due to model collapse (i.e. becoming closer to the always loss=0) or appearing nonstationarities. Thus, the metrics such as ROC or AUC scores may change drastically. To prevent their effect, this class creates windows of size window_size. After each window_size`th object, a new instance of the `metric_cls is being created. Lastly, the metrics from all windows are averaged [BGK17, BMLA18].

Parameters:

• metric_cls (class) – The metric class to be windowed.

• window_size (int) – The window size.

• ignore_nonempty_last (bool) – Whether to ignore the score of the nonempty last window. Note that the empty last window is always ignored.

Methods

__init__(metric_cls, window_size[, ...])
get()	Obtains the averaged score.
update(y_true, y_pred)	Updates the score with new true label and predicted score/label.

get()

Obtains the averaged score.

Returns:

The average score of the windows.

Return type:

float

update(y_true, y_pred)

Updates the score with new true label and predicted score/label.

Parameters:

• y_true – float The ground truth score for the incoming instance.

• y_pred – float The predicted score for the incoming instance.

Returns:

self.

Return type:

object