pysad.evaluation.WindowedMetric¶
- class pysad.evaluation.WindowedMetric(metric_cls, window_size, ignore_nonempty_last=True, **kwargs)[source]¶
A helper class to evaluate windowed metrics. The distributions of the streaming model scores often change due to model collapse (i.e. becoming closer to the always loss=0) or appearing nonstationarities. Thus, the metrics such as ROC or AUC scores may change drastically. To prevent their effect, this class creates windows of size window_size. After each window_size`th object, a new instance of the `metric_cls is being created. Lastly, the metrics from all windows are averaged [BMLA18][BGK17].
- Parameters:
Methods
__init__
(metric_cls, window_size[, ...])get
()Obtains the averaged score.
update
(y_true, y_pred)Updates the score with new true label and predicted score/label.